Method and system for redirecting driver or other vehicle operator attention

Abstract

A visual display output redirection system and method, and a visual display output control system having a visual display output redirection system and implementing the method. The method includes determining target driver attention data of a vehicle operator, where the target driver attention data indicates one or more target ranges of visual display output to be displayed to the vehicle operator; and determining visual display output data representing the visual display output adjusted based on the target driver attention data of the vehicle operator by adjusting the content of the one or more redirection ranges of the visual display output, in order to increase or reduce the likelihood of attracting visual attention of a vehicle operator towards the redirection range.

Description

Technical Field

[0001] The present invention relates to methods and systems for redirecting the attention of a driver or other operator of a vehicle, particularly by redirecting the driver's visual attention by modifying the output of the vehicle's visual display. Background Technology

[0002] A growing number of displays are being developed and introduced into vehicle applications, particularly automotive applications, to provide a better driving experience while effectively conveying more information to the driver. These displays can provide useful information for driving performance and an enhanced user experience. However, drivers, pilots, or other vehicle operators frequently gaze at, scan, or otherwise view the various visual display outputs generated by the vehicle, such as checking vehicle instruments (e.g., speed, engine temperature), assessing traffic or other elements around the vehicle, adjusting climate controls or entertainment information, making phone calls, etc., which can potentially lead to safety issues. Summary of the Invention

[0003] According to aspects of this disclosure, driver visual attention redirection technology is provided, designed to draw the attention of a driver or other vehicle operator (“vehicle operator”), i.e., the vehicle operator’s eyes, to specific content displayed on a screen. Although the description may refer to “driver” in certain circumstances, it should be understood that the teachings herein apply to any vehicle operator unless otherwise expressly stated or indicated. As used herein, “vehicle operator” refers hereinafter to a driver or an individual responsible for operating the vehicle.

[0004] According to one aspect of the invention, a method for redirecting the attention of a driver or other vehicle operator is provided. The method includes: determining target driver attention data for the vehicle operator, wherein the target driver attention data indicates one or more target ranges of visual display output to be displayed to the vehicle operator, wherein each of the one or more target ranges is a region or range of visual display output, the region or range including content determined to increase or decrease visual attention of the vehicle operator; and determining visual display output data representing a visual display output adjusted based on the target driver attention data for the vehicle operator, wherein determining the visual display output data includes adjusting the visual display output by adjusting the visual content of one or more redirected ranges of the visual display output to increase or decrease the likelihood of attracting the vehicle operator's visual attention toward one or more redirected ranges of the visual display output.

[0005] According to various embodiments, the method may also include any technically feasible combination of any one or some or all of the following features:

[0006] The current driver attention data of the vehicle operator is determined based on one or both of the sensor data and saliency map data captured at the vehicle, wherein the current driver attention data indicates the area or range of the visual display output that the vehicle operator is currently paying attention to.

[0007] Adjust the visual display output data based on the current driver attention data and the target driver attention data;

[0008] The saliency map data is determined based on the initial visual display output data, where the initial visual display output data represents the visual display output data before the visual display output is adjusted.

[0009] The initial visual display output data includes image data representing the image captured by the camera, wherein saliency map data is determined for each of multiple image ranges of the image based on the image data processing.

[0010] Based on the contrast of the image range, edge information within the image range, tonal uniqueness of the image range, texture uniqueness of the image range, and frequency components of the image range, a saliency value is assigned to each of the multiple image ranges.

[0011] The current driver attention data is determined based on sensor data, wherein the sensor data is data from a driver attention sensor installed on the vehicle and configured to capture sensor data, and wherein the sensor data provides information related to the state of the vehicle operator when the vehicle operator operates the vehicle;

[0012] The driver attention sensor is an infrared (IR) sensor configured to capture information indicating the direction of the vehicle operator's gaze;

[0013] The driver attention sensor is a device having an eye-tracking sensor and / or a head-tracking sensor, wherein the eye-tracking sensor is configured to capture eye information indicating the gaze direction of the vehicle operator, and the head-tracking sensor is configured to capture information indicating the head position of the vehicle operator, wherein the gaze direction of the vehicle driver is inferred or otherwise determined based on the vehicle operator's eye information and / or head position.

[0014] Target driver attention data is determined based on predetermined target range data, wherein the predetermined target range data indicates one or more ranges of visual display output aimed at increasing or decreasing driver attention;

[0015] One or more target ranges include one or more positive target ranges, wherein one or more positive target ranges are each area of ​​the visual display output determined to increase the visual attention of the vehicle operator, wherein the predetermined target area data includes positive target range data indicating one or more positive target ranges of the visual display output;

[0016] One or more target ranges include one or more negative target ranges, wherein one or more negative target ranges are each area of ​​the visual display output determined to reduce the visual attention of the vehicle operator, wherein the predetermined target area data includes negative target range data indicating one or more negative target ranges of the visual display output;

[0017] Target driver attention data is determined based on sensor information captured by sensors, wherein the sensor information includes sensor data or other sensor information;

[0018] The sensor information includes image data captured by a camera or infrared (IR) sensor, wherein at least one of one or more target ranges is identified as a region of the image to be displayed, wherein the region of the image is identified by a correspondence between the position of the region when it exists in the image and the position within the sensor field of view of the sensor corresponding to the position of the region;

[0019] Sensor information includes image data captured by a camera, wherein the image data is processed to detect one or more objects, wherein one or more objects are used to determine target driver attention data;

[0020] Sensor information includes depth information of one or more objects, wherein image data is processed to determine the distance of each of the one or more objects, wherein the distance of each of the one or more objects is used to determine target driver attention data; and / or

[0021] Determining the visual display output data includes generating an attention redirection map based on target driver attention data, wherein the attention redirection map indicates one or more redirection ranges, which are used to increase or decrease the visual attention of the vehicle operator, wherein one or more redirection ranges are determined based on one or more target ranges.

[0022] According to another aspect of this disclosure, a visual display output redirection system is provided, comprising: at least one processor; and a memory storing computer instructions that, when executed by the at least one processor, cause a visual display output redirection process to be performed, the process comprising: determining target driver attention data of a vehicle operator, wherein the target driver attention data indicates one or more target ranges of visual display output to be displayed to the operator, wherein each of the one or more target ranges is a region or range of visual display output, the region or range including content determined to increase or decrease visual attention of the operator; and determining visual display output data representing a visual display output adjusted based on the operator's target driver attention data, wherein determining the visual display output data includes adjusting the visual display output by adjusting the visual content of one or more redirection ranges of the visual display output to increase or decrease the likelihood of attracting the operator's visual attention toward one or more redirection ranges of the visual display output.

[0023] According to various embodiments, the visual display output redirection system can also be configured by its computer instructions to perform any or more of the features listed above related to the methods of redirecting the attention of the driver or other vehicle operator, or any technically feasible combination of some or all of the features.

[0024] According to another aspect of this disclosure, a vision display output redirection system may be included as part of a vision display output control system having a vision display configured to present vision display output for viewing by a driver or other vehicle operator. Furthermore, according to this aspect of the disclosure, the vision display output control system may also include: a camera for capturing input images as initial vision display data, wherein the initial vision display data is used to determine current driver attention data; and / or a driver attention sensor for capturing sensor data, wherein the sensor data is used to determine current driver attention data.

[0025] According to another embodiment, a method for redirecting the attention of a driver or other vehicle operator is provided. The method includes: determining target driver attention data for the vehicle operator, wherein the target driver attention data indicates one or more target ranges of visual display output to be displayed to the operator, wherein each of the one or more target ranges is a region or range of visual display output including content determined to increase or decrease visual attention of the vehicle operator; determining one or more redirection ranges based on the target driver attention data; and for each of the one or more redirection ranges, adjusting the visual display output of the redirection range using image adjustment techniques and / or graphics addition techniques.

[0026] As can be understood and appreciated from the above discussion, the various aspects disclosed herein may overlap and should not be construed as having mutually exclusive scopes unless expressly stated otherwise. Attached Figure Description

[0027] Preferred exemplary embodiments will now be described in conjunction with the accompanying drawings, wherein the same names denote the same elements, wherein:

[0028] Figure 1 This is a block diagram of a vehicle's visual display output control system according to one embodiment, which is used to redirect the driver's visual attention by adjusting the visual display output displayed on the display.

[0029] Figure 2 This is a flowchart illustrating a method for redirecting the attention of a driver or other vehicle operator according to one embodiment; and

[0030] Figure 3 This is illustrated according to one embodiment for execution. Figure 2 The block diagram and flowchart of the visual display output redirection system are described. Detailed Implementation

[0031] A system and method are provided for redirecting the attention of a driver or vehicle operator, particularly the visual attention of a vehicle operator, thereby modifying or otherwise adjusting the output of a visual display to be displayed for the vehicle operator to view in order to attract attention to a specific portion, area, or other range of the visual display output to be displayed. More specifically, according to aspects of this disclosure, one or more target ranges of the visual display output are identified, and thereafter, the content of the target range is modified to have a different appearance than before the modification, whereby the different appearance relative to the appearance of the content before modification will increase or decrease the likelihood that the vehicle operator will view the content. Thus, techniques for drawing or diverting the visual attention or focus of a vehicle operator from certain target ranges within the content to be displayed are disclosed.

[0032] According to an aspect of the present invention, a method for redirecting the attention of a vehicle driver or other vehicle operator is provided, the method comprising: determining target driver attention data of the vehicle operator; and determining visual display output data representing modified visual display output based on the target driver attention data. In at least some embodiments, the method further comprises: determining current driver attention data of the vehicle operator, wherein the visual display output data is based on the target driver attention data and the current driver attention data.

[0033] According to another aspect of the invention, the method described in the preceding paragraph is implemented by a visual display output control system, which includes at least one processor and a non-transitory computer-readable storage device storing computer instructions, which, when executed by the at least one processor, cause the visual display output control system to perform the method, thereby causing the visual display output to be adjusted according to the method, and the resulting visual display output data to be displayed for the driver to view, with the aim of increasing or decreasing the driver's attention to the target range of the visual display output.

[0034] Although the illustrated embodiments provided herein are described in detail for automotive applications, it should be understood that the vision display output control system and method are applicable to a wide range of vehicle applications, including aircraft, ships or other maritime vehicles, unmanned aerial vehicles (UAVs), etc. Therefore, unless clearly stated or otherwise provided, the discussion herein is not intended to be limited to automotive applications.

[0035] refer to Figure 1The diagram illustrates a vision display output control system 10 comprising a processing subsystem 12, a vision display 14, a driver attention sensor 16, and a camera 18. The processing subsystem 12 processes the vision display output to adjust or modify it, thereby increasing or decreasing the driver's attention to the target range of the vision display output, particularly visual attention. The vision display output control system 10 is integrated into and mounted on a vehicle V, which may be a passenger vehicle, such as a car, truck, sport utility vehicle (SUV), motorcycle, all-terrain vehicle (ATV), boat, ship, other maritime vehicle, aircraft, unmanned aerial vehicle (UAV), other aircraft, or other vehicle driven or operated by a human.

[0036] The processing subsystem 12 includes at least one processor 20 and a memory 22 storing computer instructions that, when executed by the at least one processor 20, enable the functions belonging to the processing subsystem 12 described herein to be performed. The processing subsystem 12 is configured to perform methods for redirecting the attention of a driver or other vehicle operator, such as method 200 described below. The processing subsystem 12 is mounted on a vehicle V and configured to send content to a visual display 14 such that, when displayed, the content can be seen by the driver or other operator of the vehicle V. The memory 22 is a non-transitory computer-readable memory and is accessible by the at least one processor 20, enabling the at least one processor 20 to execute computer instructions thereby performing the methods for redirecting the attention of a driver or vehicle operator.

[0037] Visual display 14 is an electronic display that visually presents content to the driver or other operator of vehicle V. Visual display 14 may be installed in the cockpit, passenger compartment, or other interior area of ​​vehicle V and configured or arranged for viewing by the driver or operator while operating vehicle V. Visual display 14 can be any of a variety of electronic displays, such as liquid crystal displays (LCDs), plasma displays, organic light-emitting diode (OLED) displays, etc. In embodiments, visual display 14 is an instrument cluster or dashboard display, an electronic display mirror, a center console display, an infotainment display, a head-up display (HUD), etc. Thus, in embodiments, visual display 14 is an electronic display with a display screen (e.g., OLED, LCD) through which light is emitted, or it may be an electronic display with a projector that projects light for display (e.g., HUD). The space where the display screen or projected content of visual display 14 is viewed visually by the driver or vehicle operator is referred to herein as the display area. Visual display 14 is communicatively coupled to processing subsystem 12 and configured to receive visual display data from processing subsystem 12 and display the visual display data when received.

[0038] The driver attention sensor 16 is used to determine driver attention data that indicates visual attention information of the driver, such as gaze direction (i.e., the direction the eyes are aiming or pointing), and in embodiments, to identify a specific location on the display area where the driver is looking or viewing. In one embodiment, the driver attention sensor 16 is an eye-tracking sensor, while in another embodiment, the driver attention sensor 16 is a head-tracking sensor. Furthermore, in embodiments, both an eye-tracking sensor and a head-tracking sensor are used in combination to determine the driver attention data.

[0039] In an embodiment where the driver attention sensor 16 is an eye-tracking sensor, the eye-tracking sensor may be, for example, an infrared (IR) sensor (such as a near-infrared (NIR)) sensor configured to acquire infrared sensor data indicating the driver's gaze direction. In one embodiment, the eye-tracking sensor is an active IR sensor that projects IR light reflected from one or both of the driver's eyes, and the reflection is captured by a receiver or sensing element of the eye-tracking sensor, whereby the sensor data is recorded and used to determine the driver's gaze direction. As used herein, "IR" or "infrared" includes near-infrared, as well as mid-infrared and far-infrared radiation.

[0040] In embodiments where the driver attention sensor 16 is a head-tracking sensor, the head-tracking sensor may be, for example, a visible light camera configured to capture images of the area where the driver is located while driving or operating the vehicle. This head-tracking sensor data is then used to determine the direction the driver's head is facing, providing information that can be used in conjunction with or in lieu of eye-tracking sensor data to infer or otherwise determine the driver's gaze direction. It should be understood that other sensors may be used to detect, infer, or otherwise determine driver attention data.

[0041] Camera 18 is a device with an image sensor for capturing image data representing an image, such as an image of an area or range outside a vehicle. Camera 18 may be a visible light camera with a visible light sensor for capturing visible light image data representing a visible light image, or it may be an infrared sensor configured to capture infrared image data representing an infrared image, such as for providing night vision. As used herein, a visible light sensor is a light sensor that captures visible light, represented as an array of pixels that together constitute a visible light image. A visible light sensor is a camera that uses a visible light color space or domain (such as RGB) to capture and represent a scene. According to embodiments, the visible light sensor is a digital camera such as one employing a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, or a Foveon sensor.

[0042] Camera 18 is mounted on the vehicle and communicatively coupled to processing subsystem 12 to provide image data to processing subsystem 12. In an embodiment, camera 18 is used to capture a video feed, which is transmitted to processing subsystem 12, whereby processing subsystem 12 processes the video feed and displays it on visual display 14.

[0043] Camera 18 can be any suitable camera type, such as a digital single-lens reflex (DSLR) camera, a mirrorless camera, a dashcam (e.g., a Garmin Dash Cam 56), a compact digital camera, a pinhole camera, etc. In one embodiment, camera 18 is an outward-facing camera. An outward-facing camera is "outward-facing" because its field of view faces and captures areas outside the vehicle, such as the road on which the vehicle is traveling, the skyline, and / or the sky. In an embodiment, visible light image data is also displayed to the driver or passengers of the vehicle, such as by displaying a real-time video stream of visible light image data captured by the outward-facing camera on the visual display 14.

[0044] Although only a single visual display 14, driver attention sensor 16, and camera 18 are shown, it should be understood that, according to embodiments, system 10 may include multiple visual displays, multiple driver attention sensors, and multiple cameras, such as multiple outward-facing cameras. Of course, in embodiments, system 10 may include various other vehicle electronics or other components.

[0045] refer to Figure 2 and Figure 3 According to one embodiment, a method 200 for redirecting the attention of a driver or other vehicle operator is shown. Figure 2 ) and the visual display output redirection system 300 configured to perform method 200. Figure 3 In at least one embodiment, method 200 is performed by the visual display output control system 10. While the steps of method 200 are described as being performed in a specific order, it should be understood that the steps of method 200 can be performed in any technically feasible order, such as performing step 220 before step 210 or performing step 220 concurrently with step 210. Furthermore, although the described and illustrated embodiments include determining current driver attention data (step 210), in other embodiments, step 210 is omitted, and in step 230, the visual display output data is determined based on target driver attention data rather than any current driver attention data.

[0046] A visual display output redirection system 300 is shown and described as an image output redirection system 300, which processes an input image 302 using a current driver attention recognition module 304, a target driver attention determination module 306, and a driver attention redirection module 308. These modules operate together to generate an output image, also referred to herein as an attention redirection image 310. Each of the current driver attention recognition module 304, the target driver attention determination module 306, and the driver attention redirection module 308 is implemented using software or computer instructions, which, when executed by a processing subsystem such as processing subsystem 12, enable the execution of the associated functions of each. At least according to an embodiment, the visual display output redirection system 300 is included as part of a visual display output control system 10, and its functions are performed by processing subsystem 12. Although by Figure 3 The thick arrows indicate the operation or processing flow, showing a specific order in which the processing can be performed. However, those skilled in the art will understand that such processing can be performed in any technically feasible order or manner, such as the processing of the target driver attention determination module 306 operating simultaneously with or before the processing of the current driver attention recognition module 304.

[0047] Method 200 begins at step 210, whereby current driver attention data of the driver operating the vehicle is determined based on one or both of sensor data and saliency map data captured at the vehicle. Current driver attention data is data indicating one or more of the following: the area or range of the visual display output that the driver is currently or is currently paying attention to, the driver's gaze direction, and the driver's head direction. In an embodiment, current driver attention data is obtained based on sensor data received at the vehicle V via attention tracking 314, which can be performed using an eye-tracking sensor (as driver attention sensor 16) that captures eye-tracking sensor data and / or a head-tracking sensor (as driver attention sensor 16) that captures head-tracking sensor data.

[0048] In some embodiments, current driver attention data is determined based on saliency map data, and the saliency map data is determined by saliency map estimation 312. When considered in conjunction with an individual's attentional use of the visual display output, the term "saliency" refers to the extent or degree to which attention is or is expected to be drawn to a particular area or portion of the visual display output relative to other areas or portions of the visual display output. For example, given a first area having a first piece of visual display output and a second area having a second piece of visual display output, the first area is considered more saliency or has a higher saliency value than the second area when the first content attracts or is expected to attract (i.e., has a higher probability of attracting) more attention or focus from the viewer. The saliency of an area of ​​visual display output can be achieved through various visual stimuli, including, for example, optical flow characteristics, motion, hue or color differences, contrast differences, color saturation, brightness, edge intensity, luminance, or transient visual characteristics (e.g., flashes or flickers, sudden changes in intensity or brightness)).

[0049] In one embodiment, saliency map data is determined based on initial visual display output data. Initial visual display output data refers to: data representing the visual display output prior to the adjustment of the visual display output in step 230; and / or data representing the visual display output displayed at a previous time (such as the time of a previous iteration of method 200). The initial visual display output includes a camera feed, thereby providing a series of images (videos) that can be transmitted from camera 18 for immediate viewing (after any appropriate or desired image processing). Within the images of the camera feed, various objects or visual artifacts can be arranged, and the visual information of said objects or artifacts is processed to identify a saliency value for each of a plurality of image ranges. For example, for each image range, a saliency value is determined and assigned to the image range based on the image range's contrast, edge information within the image range, tonal uniqueness of the image range, texture uniqueness of the image range, and frequency components of the image range.

[0050] In one embodiment, the spectral residual method is used to generate a saliency estimation map, as described by Hou, Xiaodi, and Liqing Zhang, “Saliency Detection: A Spectral Residual Approach,” IEEE Conference on Computer Vision and Pattern Recognition, 2007. This spectral residual method is designed to simulate the pre-attentional phase of human visual processing without relying on prior knowledge of objects or features. This process begins with the logarithmic spectrum of the downsampled image. The calculations capture the inherent statistical properties of natural images, which typically follow the 1 / f law in their amplitude spectrum. This is achieved by... The spectral residual is calculated by subtracting the estimated average local spectrum A(f) from the mean. Represented as:

[0051]

[0052] in

[0053] and

[0054]

[0055] Where A(f) represents the amplitude spectrum of image I(x), The operator represents the real part, and F represents the Fourier transform. The phase spectrum P(f) can be obtained by using... To calculate, where, Represents the imaginary part operator.

[0056] The spectral residuals are transformed back to the spatial domain using an inverse Fourier transform, and the result is then squared to highlight the most novel regions. The output is then smoothed with a Gaussian filter to produce the final saliency map. The transformations involved are included in the equation:

[0057] S(x)=g(x)*F -1 [exp(R(f)+P(f))] 2

[0058] Where g(x) is a Gaussian filter (such as one with σ = 8), F -1 Let P(f) denote the inverse Fourier transform, and P(f) be the phase spectrum of the image. The resulting saliency map S(x) effectively highlights native objects or regions of interest within the image. Furthermore, the saliency map can be used to create an object map through thresholding, effectively segmenting native objects from the background based on intensity thresholds. In other embodiments, other image saliency estimation techniques can be employed to generate the saliency estimation map, for example, using the techniques described in S. Ryu, B. Ham, and K. Sohn, “Contextual information-based visual saliency model,” IEEE International Conference on Image Processing 2013, Melbourne, Victoria, Australia, 2013, pp. 201-205, DOI:10.1109 / ICIP.2013.6738042.

[0059] In some embodiments, the driver's current driver attention data is determined based on sensor data from a driver attention sensor and saliency map data. For example, a driver attention sensor 16, such as an NIR active sensor, captures the driver's gaze direction, which indicates the area of ​​the visual display output that the driver is looking at (this area is referred to as the "driver gaze display area"). The saliency map data of the visual display output is then examined to provide further accuracy to the determination of the current driver attention by, for example, considering the image range of the image displayed in or near the driver gaze display area. For example, the saliency value of the image range is used to infer which specific area the driver is looking at. Method 200 continues to step 220.

[0060] In step 220, target driver attention data is determined. Target driver attention data is data indicating one or more target ranges of the visual display output to be displayed to the driver. Each of the one or more target ranges is an area or range of the visual display output that includes content determined to increase or decrease attraction to the driver's visual attention; that is, the target range is an area or range of the visual display output targeted for processing (as in step 230) to increase or decrease the driver's visual attention to that area or range. The one or more target ranges include one or more positive target ranges and / or one or more negative target ranges. One or more positive target ranges are each range of the visual display output determined to increase attraction to the driver's visual attention. Similarly, one or more negative target ranges are each range of the visual display output determined to decrease attraction to the driver's visual attention.

[0061] According to an embodiment, target driver attention data is determined based on sensor data and / or pre-determined data (such as pre-determined target range data). The target driver attention determination module 306 describes various potential types of information that can be used to determine the target driver attention data, including: a predefined (pre-determined) target range (or region of interest (ROI)) 316; data based on visible light object detection (captured by a camera) 318; data based on traffic sign detection 320 (which, when performed using image data from a visible light camera, can be considered a subtype of visible light object detection data 318); data based on a thermal camera (such as by using an infrared (IR) camera, such as a far-infrared (FIR) camera) 322; and / or data based on depth estimation 324, which can be determined from lidar and / or radar data (stereo or mono radar data).

[0062] The predetermined target range data includes: positive target range data indicating one or more positive target ranges in the visual display output; and / or negative target range data indicating one or more negative target ranges in the visual display output. Certain areas of the visual display output may be considered more important for vehicle operation. For example, in automotive applications, the road range (the range of the visual display output with the road) is more important than other ranges (such as the sky, buildings, and side ranges). The road range may be predefined (predetermined) and based on the position and orientation of the field of view of camera 18. In such embodiments, the road range is considered a positive target range, particularly a predetermined positive target range in this example, and, for example, the outer portion facing the sky may be considered a negative target range or a predetermined negative target range in this embodiment.

[0063] In one embodiment, in addition to or instead of predetermined target range data, a target attention map is determined based on sensor information (such as sensor data captured by camera 18 or other vehicle sensors, such as radar or lidar for object detection near vehicles and / or object depth estimation, thermal (or IR, such as FIR) sensors or cameras for object detection near vehicles, etc.). In one embodiment, traffic sign detection is performed, such as by capturing image data using an outward-facing camera along with image classification or other image processing techniques, enabling the identification of traffic signs within the camera's field of view, such as stop signs or speed limit signs for road vehicles.

[0064] Continuing with the previous example, object detection, such as traffic sign detection, can be used to determine the target attention map because pedestrians, cars, cracks, animals, and traffic signs are generally more important, and therefore drivers should pay more attention to these objects. Variations of object detection include bounding box object detection, pixel-by-pixel object segmentation, and object estimation. Since pedestrians, moving cars, and animals can be detected day and night regardless of weather conditions, thermal camera output can also be used to determine the target attention map. Furthermore, closer objects are generally considered important, and therefore drivers should pay attention to them; thus, such areas or ranges can be identified as positive target ranges. Depth sensors or depth estimation algorithms can also be used to estimate the distance of objects from cars or vehicles, which is used to determine the target attention map. Each set of information from pre-determined target range data, object detection, traffic signs, thermal camera data, and / or depth estimation can be combined and used to determine the target attention map. Method 200 continues to step 230.

[0065] In step 230, visual display output data representing the adjusted visual display output is determined based on the driver's target driver attention data and the current driver attention data. Determining the visual display output data involves adjusting the visual display output by modifying the target visual content of one or more target ranges of the visual display output to increase or decrease the likelihood of attracting the driver's visual attention toward one or more target ranges of the visual display output.

[0066] In at least some embodiments, determining the visual display output data includes generating an attention redirection map via attention redirection map estimation 326, thereby performing the attention redirection map estimation process based on current attention data and target driver attention data. The attention redirection map indicates one or more redirection ranges, which are used to increase or decrease the driver's visual attention. One or more redirection ranges are determined based on one or more target ranges, and in embodiments, each redirection range corresponds to a given one of the one or more target ranges; however, in embodiments or implementations, for one or more target ranges, after processing with the corresponding current driver attention data, it can be determined that such target ranges do not need to be adjusted and are therefore not considered redirection ranges because no adjustment (redirection operation) has been performed on those target ranges (e.g., ranges or areas of visual display output considered observable based on the driver's gaze direction).

[0067] The attention map estimation redirection process is used to estimate an attention redirection map based on the difference between the current attention map *ac* output by the driver to the input image or visual display and the desired target attention map *at*. Based on the attention redirection map, attention redirection 328 is performed, whereby the input image is processed to give greater attention to the positive redirection range and less attention to the negative redirection range. Greater attention to the positive redirection range (referred to as positive redirection 330) is achieved through higher contrast, more texture detail, attractive colors, higher saturation, and / or in a more direct manner (such as through attractive patterns (e.g., flashing patterns) and / or arrows). Less attention to the negative redirection range (referred to as negative redirection 332) is achieved through lower contrast, less attractive colors, lower saturation, blurring, and / or fading. In this document, "image adjustment techniques" refers to altering the appearance of an image or a portion of an image to make it more visually appealing or to draw attention to certain areas. Such image adjustment techniques may include increasing contrast (e.g., making the difference between bright and dark areas more pronounced), adding more texture detail (e.g., enhancing the appearance of surface details), using attractive colors (e.g., particularly eye-catching or pleasing colors), and / or increasing saturation (e.g., making colors more vivid). Furthermore, "graphic modification techniques" refer to adding new visual elements to an image or display to attract attention or provide additional information; this may include elements such as arrows, symbols, or patterns (e.g., flashing patterns). These techniques are commonly used in various fields, such as graphic design, advertising, and user interface design, to guide the viewer's attention and improve the user experience. In at least some embodiments, the image processing techniques are used to process the input image 302 itself, rather than by introducing new or predetermined graphics (such as arrows or text). Method 200 then ends.

[0068] Any one or more processors discussed herein can be implemented as any suitable electronic hardware capable of processing computer instructions, and can be selected according to their application. Examples of processor types that can be used include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a microprocessor, a microcontroller, etc. Any one or more non-transitory computer-readable memories discussed herein can be implemented as any suitable type of memory capable of storing data or information in a non-volatile manner and in electronic form, such that the stored data or information can be used by the processor. Memory can be any of a variety of different types of electronic memory, and a particular type can be selected based on the application using it. Examples of memory types that can be used include disk or optical disc drives, ROM (read-only memory), solid-state drives (SSDs) (including other solid-state storage such as hybrid solid-state drives (SSHDs), other types of flash memory, hard disk drives (HDDs), non-volatile random access memory (NVRAM), etc. It should be understood that any one or more computers discussed herein may include other memories, such as volatile RAM used by the processor and / or multiple processors.

[0069] It should be understood that the foregoing description pertains to one or more embodiments of the invention. The invention is not limited to the specific embodiments disclosed herein, but is defined solely by the claims. Furthermore, the statements contained in the foregoing description relate to the disclosed embodiments and should not be construed as limiting the scope of the invention or the definitions of terms used in the claims, unless the terms or phrases are explicitly defined above. Various other embodiments, as well as various changes and modifications to the disclosed embodiments, will become apparent to those skilled in the art.

[0070] As used in this specification and claims, the terms “e.g.,” “for example,” “for instance,” “such as,” and “like,” as well as the verbs “comprising,” “having,” “including,” and their other verb forms, when used with a list of one or more components or other items, are interpreted as open-ended, meaning that the list is not considered to exclude other additional components or items. Other terms should be interpreted using their broadest reasonable meaning unless they are used in a context that requires a different interpretation. Furthermore, the term “and / or” should be interpreted as inclusive or. Thus, for example, the phrase “A, B, and / or C” should be interpreted to encompass all of the following: “A,” “B,” “C,” “A and B,” “A and C,” “B and C,” and “A, B, and C.”

Claims

1. A method for redirecting the attention of a driver or other vehicle operator, comprising the following steps: Determine target driver attention data for a vehicle operator, wherein the target driver attention data indicates one or more target ranges to be output to a visual display shown to the vehicle operator, wherein each of the one or more target ranges is a region or range output by the visual display, the region or range including content determined to increase or decrease visual attention of the vehicle operator; and Determine visual display output data, the visual display output data representing a visual display output adjusted based on the target driver attention data of the vehicle operator, wherein determining the visual display output data includes adjusting the visual display output by adjusting the visual content of one or more redirection ranges of the visual display output, such that for each of the one or more redirection ranges, the likelihood of attracting the vehicle operator's visual attention toward the redirection range is increased or decreased.

2. The method of claim 1, further comprising determining current driver attention data of the vehicle operator based on one or both of sensor data and saliency map data captured at the vehicle, wherein, The current driver attention data indicates the area or range of the visual display output that the vehicle operator is currently paying attention to.

3. The method according to claim 2, wherein, The visual display output data is adjusted based on the current driver attention data and the target driver attention data.

4. The method according to claim 2, wherein, The saliency map data is determined based on initial visual display output data, wherein the initial visual display output data represents the data of the visual display output before adjustments are made to the visual display output.

5. The method according to claim 4, wherein, The initial visual display output data includes image data representing an image captured by a camera, wherein the saliency map data is determined by processing each of multiple image ranges of the image based on the image data.

6. The method according to claim 5, wherein, A saliency value is assigned to each of the plurality of image ranges based on the contrast of the image range, the edge information within the image range, the hue uniqueness of the image range, the texture uniqueness of the image range, and the frequency component of the image range.

7. The method according to claim 2, wherein, The current driver attention data is determined based on the sensor data, wherein the sensor data is data from a driver attention sensor installed on the vehicle and configured to capture the sensor data, and wherein the sensor data provides information related to the state of the vehicle operator when the vehicle operator operates the vehicle.

8. The method according to claim 7, wherein, The driver attention sensor is an infrared (IR) sensor configured to capture information indicating the direction of the vehicle operator's gaze.

9. The method according to claim 8, wherein, The driver attention sensor is a device having an eye-tracking sensor and / or a head-tracking sensor, the eye-tracking sensor being configured to capture eye information indicating the direction of the vehicle operator's gaze, and the head-tracking sensor being configured to capture information indicating the position of the vehicle operator's head, wherein the direction of the vehicle operator's gaze is inferred or otherwise determined based on the vehicle operator's eye information and / or head position.

10. The method according to claim 1, wherein the target driver attention data is determined based on predetermined target range data, wherein, The predetermined target range data indicates one or more ranges of the visual display output aimed at increasing or decreasing driver attention.

11. The method according to claim 1, wherein, The one or more target ranges include one or more positive target ranges, wherein the one or more positive target ranges are each range of the visual display output determined to increase the visual attention of the vehicle operator.

12. The method according to claim 11, wherein, The target driver attention data is determined based on predetermined target range data, wherein the predetermined target range data includes positive target range data indicating one or more positive target ranges output by the visual display.

13. The method according to claim 1, wherein, The one or more target ranges include one or more negative target ranges, wherein the one or more negative target ranges are each range determined to reduce the visual display output that attracts the visual attention of the vehicle operator.

14. The method according to claim 13, wherein, The target driver attention data is determined based on predetermined target range data, wherein the predetermined target range data includes negative target range data indicating one or more negative target ranges output by the visual display.

15. The method according to claim 1, wherein, The target driver attention data is determined based on sensor information captured by sensors, wherein the sensor information includes the sensor data or other sensor information.

16. The method according to claim 15, wherein, The sensor information includes image data captured by a camera or infrared (IR) sensor, wherein at least one of the one or more target ranges is identified as a region of the image to be displayed, wherein the region of the image is identified by a correspondence between the position of the region when it exists within the image and the position within the sensor field of view of a sensor corresponding to the position of the region.

17. The method according to claim 15, wherein, The sensor information includes image data and / or depth information of one or more objects, wherein the one or more objects are used to determine the target driver attention data, wherein the image data is data processed to detect the one or more objects, and wherein the depth information indicates the distance of each of the one or more objects.

18. The method according to claim 1, wherein, Determining the visual display output data includes generating an attention redirection map based on the target driver attention data, wherein the attention redirection map indicates one or more redirection ranges, and the one or more redirection ranges are used to increase or decrease the visual attention of the vehicle operator, wherein the one or more redirection ranges are determined based on the one or more target ranges.

19. A visual display output redirection system, comprising: At least one processor; and a memory storing computer instructions, which, when executed by the at least one processor, cause a visual display output redirection process to be performed, the process including: Determine target driver attention data for a vehicle operator, wherein the target driver attention data indicates one or more target ranges to be output to a visual display shown to the operator, wherein each of the one or more target ranges is a region or range output by the visual display, the region or range including content determined to increase or decrease visual attention of the operator; and Determine visual display output data, the visual display output data representing a visual display output adjusted based on the operator's target driver attention data, wherein determining the visual display output data includes adjusting the visual display output by adjusting the visual content of one or more redirection ranges of the visual display output in order to increase or decrease the likelihood of attracting the operator's visual attention toward the one or more redirection ranges of the visual display output.

20. A method for redirecting the attention of a driver or other vehicle operator, comprising the following steps: Determine target driver attention data for a vehicle operator, wherein the target driver attention data indicates one or more target ranges to be output to a visual display to be shown to the vehicle operator, wherein each of the one or more target ranges is a region or range output by the visual display, the region or range including content determined to increase or decrease the visual attention of the vehicle operator. Based on the target driver attention data, determine one or more redirection ranges; and For each of the one or more redirection ranges, the visual display output of the redirection range is adjusted using image adjustment techniques and / or graphics addition techniques.

Images