Retina perception-based adaptive display adjustment system and method for vehicle-mounted ar-hud

Abstract

The application discloses a kind of based on retinal perception's vehicle-mounted augmented reality head-up display (AR-HUD) adaptive display adjustment system and method. System includes: retinal perception parameter extraction, driving instruction response analysis, display light environment acquisition analysis and dynamic adaptive display optimization module. The retinal perception threshold of user is obtained by contrast sensitivity test before driving, and objective physiological parameters such as pupil diameter, blink rate and fixation stability in a preset time window are extracted during driving, and the comprehensive fatigue index F is calculated;Combined with the data of vehicle-mounted light sensor, the display area brightness distribution model is established, the brightness and contrast of AR-HUD picture are adaptively adjusted and safety limiting based on F, to ensure that the key information reaches the minimum identifiable contrast threshold and suppresses the over-bright area. Compared with the fixed display adjustment scheme, the present application can improve the readability of information and reduce the subjective fatigue score in different light environments, and improve the adaptability and viewing comfort of vehicle-mounted augmented reality head-up display.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-mounted human-computer interaction and augmented reality display technology, specifically relating to a system and method for adaptively adjusting the display of vehicle-mounted AR-HUD images based on real-time acquisition of the driver's retinal perception state and external lighting information to improve information visibility and driving safety. Background Technology

[0002] Augmented Reality Head-Up Displays (AR-HUDs) have been applied in in-vehicle human-machine interaction, projecting virtual information such as navigation, vehicle speed, and driver assistance prompts onto the driver's field of vision and optically overlaying it with the real road scene. This reduces the frequency with which the driver's eyes leave the road, improving information acquisition efficiency and driving safety. However, current AR-HUD technology still has shortcomings in adaptive display adjustment:

[0003] First, it has poor adaptability to environmental changes. When most displays adjust global brightness based solely on a single point of ambient light, the displayed information becomes either too bright and glaring or too dark and difficult to read, thus becoming a new source of visual interference.

[0004] Secondly, it ignores individual and state-related differences. Current technology treats drivers as a "standard model" with constant visual ability, completely ignoring the differences in visual perception between different drivers, and even among the same driver in different states. In fact, a driver's visual contrast sensitivity changes dynamically due to factors such as fatigue, age, and health status. Furthermore, making "visible optimizations" to the display based on the user's actual perception can avoid redundant display information and wasted resources on ineffective image enhancement.

[0005] Finally, there is a lack of a closed-loop human-computer interaction mechanism. Existing systems often have preset fixed adjustment strategies and lack a closed-loop evaluation of whether the displayed information is effectively and quickly understood by the driver. Therefore, they cannot optimize themselves based on real interaction performance data, making it difficult to achieve an intelligent and human-centered interactive experience. Summary of the Invention

[0006] Technical challenge: Under complex lighting conditions and varying driving loads, how to generate appropriate adaptive display adjustment commands for AR-HUD images based on the driver's retinal perception state and the display lighting environment, while ensuring minimum visibility and safety limits.

[0007] Technical solution: To solve the above technical problems, the present invention adopts the following technical solution:

[0008] A vehicle-mounted AR-HUD adaptive display adjustment system based on retinal perception, characterized in that it includes:

[0009] Retinal perception parameter extraction module: used to acquire the driver's pupil diameter, blink rate and gaze trajectory through eye movement sensors and calculate a set of visual state parameters within a preset time window. The set of parameters includes at least a contrast perception threshold and a comprehensive fatigue index.

[0010] Driving command response parsing module: used to collect human-machine interaction events (brake, steering wheel, turn signal) and their timestamps, calculate the display-response latency and accuracy of each command, and output the current response sensitivity;

[0011] Display light environment acquisition and analysis module: used to acquire the ambient illuminance within the driver's field of vision and establish an equivalent illuminance distribution model by combining it with the panel's self-illuminating brightness;

[0012] Dynamic adaptive display optimization module: used to couple the output parameters of the above modules to generate brightness and contrast control commands, and output the control signals to the AR-HUD controller.

[0013] Preferably, the retinal sensing parameter extraction module includes:

[0014] Eye physiological characteristic monitoring unit: used to acquire pupil diameter, blink rate and fixation point trajectory in real time and synchronize them to the same timestamp;

[0015] Retinal perception parameter calculation unit: used to obtain the user's contrast threshold function through contrast sensitivity test, and calculate the comprehensive fatigue index based on pupil diameter, blink rate baseline value and gaze fatigue index.

[0016] Preferably, the driving command response parsing module includes:

[0017] Response data acquisition unit: used to record the operation event type and timestamp of the brake pedal, steering wheel and turn signal components, and to obtain the presentation time of the corresponding command on the AR-HUD;

[0018] Command Response Analysis Unit: Used to calculate the response latency and accuracy of each command, forming driving response sensitivity.

[0019] This invention also provides a retina-sensing-based adaptive display adjustment method for in-vehicle AR-HUD based on the system, comprising the following steps:

[0020] During the contrast sensitivity test, the user's pupil diameter was collected. blink rate and gaze trajectory ( , And aligned with the timing of stimulus presentation;

[0021] Based on the results of the contrast sensitivity test, we can obtain the user's sensitivity to different directions. and spatial frequency The just-perceptible contrast threshold under combination At the same time, the baseline is obtained. , Compared with baseline ;

[0022] Record the presentation time of command information in AR-HUD Record the type and the corresponding user operation time. type;

[0023] calculate and response sensitivity ;

[0024] Calculation within a fixed time window and gaze drift speed And calculate the comprehensive fatigue index. ;

[0025] Collect ambient illuminance in the AR-HUD area With the brightness of the panel's self-illumination Establish an equivalent brightness distribution model ;

[0026] Generate brightness and contrast adjustment instructions It then performs visibility and safety limiting to obtain the output command.

[0027] Preferably, the contrast sensitivity test includes converting the display information into a corresponding spatial frequency conversion formula:

[0028]

[0029] in, The viewing angle corresponding to the complete cycle of a set of bright and dark stripes, displayed in degrees.

[0030] Preferably, the minimum visible brightness threshold is determined by the viewing direction. The spatial frequency 𝑓 is calculated as follows:

[0031]

[0032] in, and This represents the panel's maximum brightness and maximum contrast. To view the direction Contrast threshold at spatial frequency φ.

[0033] Preferably, response sensitivity Calculate using the following formula:

[0034] ,

[0035] , ;

[0036] Preferably, the comprehensive fatigue index The result is obtained by weighting the pupil, blink, and fixation sub-indices:

[0037]

[0038]

[0039]

[0040]

[0041] ,

[0042] , ,

[0043] For preset time windows, Baseline pupil diameter, Baseline blink rate, For baseline gaze dispersion, γ represents the baseline average drift velocity, and γ, a, b, and c are weighting coefficients.

[0044] Preferably, the equivalent brightness distribution model is:

[0045]

[0046] in, This refers to the light transmittance of the AR-HUD.

[0047] Preferably, the brightness and contrast adjustment and limiting formulas include:

[0048]

[0049] )

[0050]

[0051]

[0052] in, = For the above The spatiotemporal average value, This is a limiting function to ensure that the output value is within a safe range.

[0053] Preferably, when a sudden change in ambient illuminance is detected, the brightness and contrast of the display screen are adjusted immediately, and the sudden change criterion is satisfied.

[0054]

[0055] in, The illuminance at the current moment, The preset threshold for judging sudden changes in illuminance is given, and the unit is lux.

[0056] Beneficial effects: This invention introduces adaptive display control based on user-personalized visual perception parameters and visual fatigue state to achieve dynamic adjustment of AR-HUD brightness and contrast: The system calculates a comprehensive fatigue index based on physiological quantities such as pupil diameter, blink rate and gaze stability, and establishes an equivalent brightness model by combining ambient light sensor and panel self-illumination, and finally generates an adjustment value that can simultaneously meet the minimum visual threshold and physical upper limit.

[0057] Thus, under complex lighting conditions and different driving loads, it avoids both excessive darkness causing information to be unrecognizable and excessive brightness causing glare and increased visual load, thereby ensuring the stable visibility and recognizability of key information and improving comfort, driving safety and system reliability in long-term use scenarios. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the online adaptive display adjustment process for the vehicle-mounted AR-HUD of the present invention.

[0059] Figure 2 This is a schematic diagram illustrating the sensitivity testing and threshold calibration process.

[0060] Figure 3 This is a heatmap of contrast sensitivity under various ambient illuminance levels. The top four groups represent the contrast sensitivity of the human eye to a desktop 4K OLED display in front of a high-reflectivity white wall at 30 cd / m². 2 The following four sets of test results represent the contrast sensitivity test results in front of a black wall with low reflectivity. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings.

[0062] Example 1

[0063] This embodiment addresses the entire driving process, and the flowchart for adaptively adjusting the brightness and contrast of the head-up display (HUD) image based on eye-tracking physiological monitoring, human-computer interaction analysis, and ambient light information collection is as follows: Figure 1As shown. The control software runs on an automotive-grade processor, and all sensor data is synchronized according to the vehicle's unified time base. The specific steps include:

[0064] (1) After the vehicle system is initialized, it enters the S101 pre-driving retinal perception parameter extraction stage:

[0065] (1.1) After the system starts, the central processing unit (MCU) wakes up the infrared eye-tracking camera in front of the driver's seat and turns on the ambient light sensor (ALS). After the driver clicks "Start Test", the pupil diameter sequence is recorded with that moment as the "start event" timestamp. Blink event timestamp set, gaze point trajectory ( , Before the "end event" arrives, all data is aligned with the vehicle's clock and incorporated into the circular buffer.

[0066] Preferably, the frame rate of the infrared eye-tracking camera is ≥60 fps and the ALS sampling frequency is ≥50 Hz.

[0067] (1.2) During data acquisition, the MCU operates within a sliding time window. Real-time calculation of average pupil diameter blink rate , observation and distribution Mean gaze drift velocity The missing frame rate is recorded, and linear interpolation is used for missing frames.

[0068] Preferably, the formula for calculating gaze dispersion is as follows: The formula for calculating the average gaze drift velocity is: .

[0069] (2) After collecting the driver's initial visual perception parameter set, proceed to the S102 driving command response parsing process:

[0070] (2.1) The HUD displays navigation, speed limit, and other instructions based on road conditions. The system generates a presentation timestamp for each instruction. Simultaneously, it monitors the operation channels bound to the command (such as steering lever, brake / accelerator pedal, etc.) and captures the response timestamp. .

[0071] (2.2) The controller operates within a preset matching time window The system pairs each response action with a command and determines the response status as "correct / incorrect" based on the command type. For example, a left turn prompt indicating a detected left lever response is considered correct; all others are considered incorrect. Accuracy rate. Using a fixed sample size Rolling statistics are used; if no corresponding response is found within the matching time window, it is recorded as "no response". The output response sensitivity is calculated.

[0072] , .

[0073] Preferably, matching time window Requires ≥2 s, sample size .

[0074] (3) The display light environment analysis of the S103 field of vision light environment analysis module is synchronized with the driver's visual state monitoring process:

[0075] (3.1) The system directly generates the spatial illuminance matrix based on ALS-acquired data. The panel self-illumination brightness configuration in the system files. Synthetic equivalent brightness distribution:

[0076] .

[0077] Preferably, This refers to the light transmittance of the AR-HUD panel material in the initial factory specifications of the vehicle.

[0078] (3.2) Determination of sudden changes in light environment based on time period Preset absolute threshold And conforms to

[0079] .

[0080] Preferably, Different values ​​can be selected for daytime and nighttime driving periods, such as tunnels and nighttime. =40 lux, suitable for use under strong daylight. =150 lux; Time period can be selected =1 s. If the above formula is met for at least two consecutive cycles, then if the result of "strong light change" is "yes", proceed to S104 and perform a fast adjustment first; if there is no strong light change, return directly to S102.

[0081] (4) Enter the S104 display screen brightness and contrast adaptive module:

[0082] (4.1) The reference brightness is the spatiotemporal average of the equivalent brightness. First, perform a linear callback on the entire image:

[0083] .

[0084] Preferably, the fatigue index can be calculated as follows: Limit the values ​​to the range [0,1], and assign weights. Otherwise, .

[0085] (4.2) Set of key information elements (Such as navigation steering, alarms) Ensure minimum visibility and safety limits:

[0086] Minimum visible brightness is calculated from a contrast threshold obtained through personal calibration.

[0087]

[0088] in This is the maximum brightness of the panel. This is the panel's maximum contrast ratio. To ensure that the contrast thresholds are accurately detected, they all originate from the contrast sensitivity measurement and threshold calibration process of this invention. Figure 2 , S201–S207) income;

[0089] Brightness output limiting

[0090]

[0091] Then, the local contrast ratio is calculated based on the reference brightness and limited:

[0092]

[0093]

[0094] After limiting, pixel-level adjustment commands are generated and sent to the HUD driver unit.

[0095] (4.3) When S103 determines that “strong light change” is true, the control cycle is temporarily shortened by one time to quickly align brightness and contrast. The normal cycle is restored after the scene stabilizes.

[0096] (4.4) When the ALS or eye-tracking camera is in an abnormal state (e.g., transmission timeout, packet loss rate > 60%) or the personal threshold database is missing, let Adjustment only executes and The brightness is limited, the display contrast is output according to the maximum contrast allowed by the panel, and the fault code and timestamp are recorded. It will automatically exit after the connection is restored.

[0097] (5) Steps (1)-(4) are continuously cycled during vehicle operation. After each intense light change event, the system will generate an event summary (trigger cause, peak value, etc.). (Adjustment of maximum amplitude and duration) and key indicators (minimum) ,maximum The response time is written to the runtime log for subsequent calls and legal compliance audits.

[0098] The above embodiments are described The parameters can be calibrated within a limited range according to the vehicle model and scenario, without changing the essence of the method. This embodiment does not lock the hardware brand and interface; as long as the sampling accuracy / time base synchronization is met, it can be considered an equivalent implementation.

[0099] Example 2

[0100] This embodiment is used when the vehicle is stationary or under safe conditions, such as... Figure 2 The diagram shows the individualized calibration of contrast sensitivity and minimum visible brightness threshold for the driver. The calibration results will serve as the lookup table input for the online adaptive adjustment in Example 1, ensuring that different drivers can obtain a sufficiently visible and unexposed HUD presentation at different orientations and spatial frequencies.

[0101] (1) Initialization of display stimulus parameters and preparation for display consistency based on step S201

[0102] (1.1) Set the initial spatial frequency set With display of orientation set .

[0103] Preferably, the spatial frequency set can be selected as follows: ={0.5, 1, 2, 4, 8} cpd, displays the available location set. ={C, L, R, U} correspond to the center, left side of the center, right side of the center, and top, respectively.

[0104] (1.2) In addition to the default spatial frequency range mentioned above, the main stroke angle width of the system's built-in display symbols also needs to be adjusted. (Unit: degrees) Convert to additional spatial frequency levels:

[0105] .

[0106] (1.3) Read the HUD grayscale-brightness lookup table from the system's factory settings and record the panel's peak brightness. With maximum contrast Simultaneously calculate Michelson contrast ratio :

[0107] .

[0108] (1.4) Present the fixation point at the center of the field of view and display the corresponding directional symbols in the areas where stimuli may appear to define the display range of each direction.

[0109] Preferably, if a departure from the fixation point or a large saccade (instantaneous velocity > 60) is detected, The current trial is invalidated and presented again.

[0110] (2) Environmental illuminance acquisition and stability determination based on step S202:

[0111] (2.1) The ALS sampling frequency must be ≥50 Hz, which can be determined according to the light acquisition illuminance. Figure 3 A cross-illuminance threshold library was established based on the conversion relationship between different ambient illuminance levels.

[0112] (3) Randomized sinusoidal grating presentation based on step S203 (single trial):

[0113] (3.1) Randomly select a set from the unfinished spatial frequency-display azimuth set phase A sinusoidal grating is presented within a fixed window.

[0114] Preferably, the duration for which the stimulus can display the maximum contrast is 3 seconds, and the masking interval between stimuli can be 2 seconds to avoid visual aftereffects.

[0115] (3.2) Initial contrast Set to 0.1, the step size is increased in a stepwise manner to ensure search efficiency and stability. (3.3) During this period, the driver uses the directional keys to navigate within the response judgment time window. Perform a "visible / invisible" judgment within.

[0116] Preferably, To avoid prolonged eye strain, if the viewer loses focus or blinks and obstructs the view, the current attempt is deemed invalid and rewritten.

[0117] (4) Threshold optimization and convergence record based on steps S204–S205 (for a single record) The contrast is changed using a stepwise method during stimulus display until the stimulus contrast is inverted at least four times and the processor receives a correct response from the user. The just-perceptible contrast threshold is denoted as .

[0118] Preferably, it can be taken =2 dB, the step method adopts a 1-up / 2-down strategy.

[0119] (5) Combinatorial coverage and sequential strategy based on step S206:

[0120] (5.1) If there are still unfinished projects Combine the elements, randomly select the next group and return it to S203; add completed combinations to the "Completed Set".

[0121] (5.2) To reduce the learning and anticipation effects, the orientation must not be repeated in any three consecutive trials; the same The interval between two trials is ≥ 5 trials.

[0122] (6) Based on the threshold curve output in step S207, convert the brightness threshold to a complete individualized contrast threshold curve:

[0123]

[0124] Calculate the minimum visible brightness threshold based on the panel capabilities (provided to S104 of Example 1):

[0125]

[0126] Optional output orientation visibility weights (used for online prioritization or contrast budget allocation):

[0127] .

[0128] Preferably, if the gaze loss rate is >50% in all the above steps, the subject is prompted to check the eye-tracking acquisition device and retest.

Claims

1. A vehicle-mounted AR-HUD adaptive display adjustment system based on retinal perception, characterized in that, include: The retinal perception parameter extraction module is used to acquire the driver's pupil diameter, blink rate, and fixation trajectory via an eye-tracking sensor, and calculate a set of visual state parameters within a preset time window. This parameter set includes at least a just-perceptible contrast threshold. With the overall fatigue index, and These represent the viewing direction and spatial frequency, respectively. Driving command response parsing module: used to collect human-computer interaction events and their timestamps, calculate the display-response latency and accuracy of each command, and output the current response sensitivity; Display light environment acquisition and analysis module: used to acquire ambient illuminance within the driver's field of vision. Combined with the panel's self-illuminating brightness Establish an equivalent luminance distribution model: ； in, For AR-HUD light transmittance; Dynamic adaptive display optimization module: used to optimize display based on comprehensive fatigue index. Brightness distribution model And just noticeable contrast threshold Generate brightness adjustment instructions and contrast adjustment commands Output instructions are generated using the clip function. and The output command is sent to the AR-HUD controller; the specific formula is: ； )； ； ； in, = Brightness distribution model The spatiotemporal average value; , and These represent the panel's maximum brightness and maximum contrast.

2. The system according to claim 1, characterized in that, The retinal sensing parameter extraction module includes: Eye physiological characteristic monitoring unit: used to acquire pupil diameter, blink rate and fixation point trajectory in real time and synchronize them to the same timestamp; Retinal perception parameter calculation unit: used to obtain the user's just-perceptible contrast threshold through contrast sensitivity testing. The comprehensive fatigue index was calculated based on pupil diameter, baseline blink rate, and gaze fatigue index.

3. The system according to claim 1, characterized in that, The driving command response parsing module includes: Response data acquisition unit: used to record the operation event type and timestamp of the brake pedal, steering wheel and turn signal components, and to obtain the presentation time of the corresponding command on the AR-HUD; Command Response Analysis Unit: Used to calculate the response latency and accuracy of each command, forming driving response sensitivity.

4. A retinal-sensing-based adaptive display adjustment method for in-vehicle AR-HUD based on the system of any one of claims 1-3, characterized in that, Includes the following steps: S1: Collect the user's pupil diameter during the contrast sensitivity test. blink rate and gaze trajectory ( , And aligned with the timing of stimulus presentation; S2: Based on the test results of S1, obtain the user's preferences for different directions. and spatial frequency The just-perceptible contrast threshold under combination At the same time, the baseline pupil diameter was obtained. Baseline blink rate Baseline gaze dispersion Average drift velocity with baseline ; S3: Record the presentation time of command information in AR-HUD Record the type and the corresponding user operation time. With type; S4: Accuracy of operation calculated based on the records of S3 And calculate the user's response sensitivity. ; S5: Calculate the average pupil diameter within a fixed time window. Blink rate (BR) and fixation dispersion and average gaze drift speed And calculate the comprehensive fatigue index. ; S6: Collect ambient light levels in the AR-HUD area With the brightness of the panel's self-illumination Establish an equivalent brightness distribution model ,in, For AR-HUD light transmittance; S7: Based on S2 S5 And S6 Generate brightness and contrast adjustment instructions It then performs visibility and safety limiting to obtain the output command. The output command is sent to the AR-HUD controller; the specific formula is: ； )； ； ； in, = Brightness distribution model The spatiotemporal average value; , and These represent the panel's maximum brightness and maximum contrast.

5. The method according to claim 4, characterized in that, Step S1, the contrast sensitivity test, includes converting the display information into the corresponding spatial frequency conversion formula: ； in, The viewing angle corresponding to the complete cycle of a set of bright and dark stripes to display information.

6. The method according to claim 4, characterized in that, Minimum visible brightness threshold by viewing direction The spatial frequency 𝑓 is calculated as follows: ； in, and This represents the panel's maximum brightness and maximum contrast. To view the direction Contrast threshold at spatial frequency φ.

7. The method according to claim 4, characterized in that, Response sensitivity in step S4 Calculate using the following formula: ， ； , To preset the matching time window, To improve operational accuracy, and These are the weighting coefficients.

8. The method according to claim 4, characterized in that, The comprehensive fatigue index in step S5 The result is obtained by weighting the pupil, blink, and fixation sub-indices: ； ； ； ； ， ； ， ， ； The preset time window is defined by γ, a, b, and c, which are weighting coefficients.

Images