Adaptive fixation point display method and system based on content awareness and user calibration

Abstract

The invention discloses a self-adaptive fixation point display method and system based on content awareness and user calibration, and the method comprises the steps: carrying out the calibration of a visual attention range when the system is started for the first time, and obtaining the individualized initial fixation area parameters of a user; the position of a fixation point is obtained in real time through an eye movement tracking module, and eyeball movement states are classified; identifying a content semantic category of the current display picture; according to the content semantic category and the initial fixation area parameters, dynamically determining geometric parameters of the high-fidelity rendering area; executing rendering operation for reducing power consumption outside the high-fidelity rendering area; and monitoring content change in the area, and when key non-periodic change is detected, performing temporary vision enhancement operation on the changed area. According to the method, the clear display area can be dynamically adjusted according to the screen content and the personalized visual features of the user, significant energy saving is achieved, meanwhile, it is ensured that the user does not miss important information through an intelligent visual enhancement mechanism, and display efficiency, power consumption control and user experience are considered.

Description

Technical Field

[0001] This invention relates to the field of display control technology, and in particular to an adaptive gaze point display method and system based on content awareness and user calibration. Background Technology

[0002] Currently, eye-tracking technology has been applied in multiple fields. At the basic algorithm level, existing technologies have improved the accuracy and robustness of gaze prediction by fusing facial orientation, eye images, and ambient light information (such as glare) using machine learning models. At the application level, some solutions have implemented gaze-based partitioned rendering. For example, in virtual reality (VR) scenarios, high-resolution rendering is performed on the area around the user's gaze center, while the level of detail in the surrounding areas is reduced to save the computing power of the graphics processing unit (GPU). In addition, research is exploring the use of eye-tracking data for attention state monitoring, such as assessing the cognitive load of drivers in driving scenarios.

[0003] Existing technologies have significant shortcomings and cannot meet the needs of general-purpose computing devices (such as laptops and tablets) to achieve both extreme energy efficiency and a lossless information experience: Rigid Strategy: Existing partitioned rendering schemes mostly use fixed or simply scaled viewing areas, failing to dynamically adjust the geometry of high-fidelity areas according to the semantics of screen content (such as text, images, and UI interfaces), resulting in excessive interference during reading or loss of key details when browsing images.

[0004] Information obscuring risk: Permanently degrading non-focused areas (such as blurring or darkening) will completely obscure new messages, error messages, or important content changes appearing in that area. Users are very likely to miss key information, resulting in operational errors or a decline in experience.

[0005] Lack of personalized adaptation: The existing solution does not take into account the individual differences of users (such as age, vision, and usage habits), nor does it provide a scientific initialization process and a convenient manual adjustment mechanism, resulting in a "one-size-fits-all" system that cannot truly meet user needs.

[0006] Application scenario limitations: Mainstream foveated rendering technology is mainly aimed at closed, high-computing scenarios such as VR / AR. Its goal is to reduce GPU load, rather than to optimize pixel power consumption at the hardware level for self-emissive displays such as OLED. Summary of the Invention

[0007] The purpose of this invention is to provide an adaptive gaze point display method and system based on content awareness and user calibration, which significantly reduces screen power consumption while ensuring that users have a focused, efficient and information-free visual experience by dynamically adapting to key changes in the user's attention range and intelligent vision enhancement.

[0008] The technical solution adopted in this invention is: An adaptive gaze point display method based on content awareness and user calibration is characterized by the following steps: S1. When the function is first enabled, perform a visual attention range calibration test to obtain and store the user's individualized initial gaze area parameters; S2. The eye-tracking module acquires the user's gaze point position in real time and identifies the semantic category of the content displayed on the screen. S3. Dynamically determine the geometric parameters of the high-fidelity rendering region based on the content semantic category and the initial gaze area parameters; S4. Perform power consumption optimization operations on the display area outside the high-fidelity rendering area; S5. Monitor changes in content within the power consumption optimization operation area. When a key change event that meets the preset criteria is detected, perform a temporary visual enhancement operation on the area where the key change event occurred.

[0009] Furthermore, the specific process of step S1 is as follows: The test screen is displayed, which includes a central gaze cue point and multiple interactive target elements randomly distributed in the area surrounding the central gaze cue point. Receive user clicks on interactive target elements via input devices and record the position coordinates of successfully clicked target elements; Based on the recorded location coordinates, the personalized attention range boundary centered on the central gaze cue point is calculated using a geometric fitting algorithm; The parameters of the personalized attention range boundary are stored as initial gaze area parameters.

[0010] Furthermore, in step S2, the process of obtaining the user's gaze point position includes real-time classification of the user's eye movement data: Based on a preset velocity-acceleration threshold model or machine learning classifier, eye movement states are classified into fixation events or saccadic events.

[0011] Furthermore, step S3 is only executed when the current eye movement state is classified as a gaze event; when it is classified as a saccade event, the system maintains the position of the high-fidelity rendering area unchanged, or uses a prediction algorithm based on the historical gaze point trajectory to smoothly transition the position of the high-fidelity rendering area.

[0012] Furthermore, the steps in step S5 for monitoring content changes include: Identify whether the detected content changes have periodic characteristics; If the content changes are identified as having periodic characteristics, then temporary visual enhancement operations are suppressed.

[0013] Furthermore, for non-periodic content changes, priority is assessed based on the semantic information of the content changes, and they are divided into high priority, medium priority, and low priority. Perform temporary visual enhancements only on high-priority and medium-priority content changes.

[0014] Furthermore, in step S4, performing a rendering operation to reduce power consumption includes: for OLED or Micro-LED screens, turning off the brightness of pixels outside the high-fidelity rendering area; for LCD screens, applying a filter to reduce visual clarity outside the high-fidelity rendering area.

[0015] Furthermore, the duration of the visual enhancement operation is between 300 milliseconds and 800 milliseconds.

[0016] This invention discloses an adaptive gaze point display system based on content awareness and user calibration, used to implement the above-mentioned method, the system comprising: The eye-tracking module is used to acquire the user's eye movement data in real time, and classify the eye movement state through a preset velocity-acceleration threshold model or machine learning classifier, and output the judgment result of fixation event or saccade event; The content analysis module is used to identify the semantic category of the content currently displayed and to monitor content changes in non-focused areas; The decision engine receives the judgment results of gaze events and dynamically determines the geometric parameters of the high-fidelity rendering area based on the semantic category of the content and the user's individualized initial gaze area parameters. During saccade events, it outputs instructions to maintain the position of the high-fidelity rendering area unchanged, or to smoothly transition the position of the high-fidelity rendering area based on the historical gaze point trajectory through a prediction algorithm. The partition rendering control module receives control commands from the decision engine, performs power optimization operations on non-high-fidelity rendering areas, and performs temporary visual enhancement operations on specified changing areas.

[0017] Furthermore, the eye-tracking module includes a device or software system capable of outputting the position information of the user's gaze point in the screen coordinate system.

[0018] The present invention adopts the above technical solution and has the following beneficial technical effects: 1. Content-Aware Dynamic Attention Zone: The system can intelligently identify the type of currently displayed content (such as text, images, and UI interfaces) and dynamically adjust the high-fidelity rendering area (attention zone) accordingly. This overcomes the problem of rigid attention zone strategies in existing technologies.

[0019] 2. Change-driven temporary visual enhancement mechanism: The system continuously monitors content changes in non-focused areas. Once a preset key event is detected (such as a new notification, error pop-up, or data update), the changed area will be automatically and briefly highlighted to ensure that users do not miss important information, effectively solving the risk of information occlusion.

[0020] 3. Dual-track user attention model: Upon initial use, the system guides the user through a brief visual attention range calibration test to establish a personalized initial baseline. Simultaneously, an intuitive user-customizable interface allows users to manually adjust or override system settings at any time. This perfectly balances system intelligence with user control.

[0021] 4. Deep integration and energy saving: For OLED / MicroLED self-emissive screens, deep energy saving is achieved through hardware-level pixel shutdown; for LCD and other backlit screens, it degenerates into software-level visual filters, ensuring the universality of the solution across all platforms and maximizing the energy-saving effect. Attached Figure Description

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 A schematic diagram for performing a visual attention range calibration test; Figure 2 This is a schematic diagram showing the high-fidelity rendered area in text mode. Figure 3 This is a schematic diagram showing the high-fidelity rendering area in image mode; Figure 4 A diagram illustrating the application of temporary visual enhancements to areas outside the high-fidelity rendering region; Figure 5 This is an architecture diagram of the system of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0024] This invention discloses an adaptive gaze point display method based on content awareness and user calibration, comprising the following steps: S1. Upon initial activation of the function, a visual attention range calibration test is performed to acquire and store the user's individualized initial gaze area parameters; the specific process is as follows: The test screen is displayed, which includes a central gaze cue point and multiple interactive target elements randomly distributed in the area surrounding the central gaze cue point. The user is guided to keep their head still while the system receives clicks from the user on interactive target elements via input devices (such as a mouse, touchpad, or keyboard) and records the position coordinates of the successfully clicked target element. • Based on the recorded position coordinates, the personalized attention range boundary centered on the central gaze cue point is calculated using a geometric fitting algorithm (e.g., least squares fitting of an ellipse); Store the parameters of the personalized attention range boundary (such as semi-major axis, semi-minor axis, and yaw angle) as the initial gaze area parameters.

[0025] S2. The eye-tracking module acquires the user's gaze point position in real time and identifies the semantic category of the currently displayed content; the process of acquiring the user's gaze point position includes real-time classification of the user's eye movement data: Based on a preset velocity-acceleration threshold model or machine learning classifier, eye movement states are classified into fixation events or saccadic events.

[0026] The system performs the dynamic determination of the high-fidelity rendering region in step S3 only when the current eye movement state is classified as a gaze event. When it is classified as a saccade event, the system maintains the position of the high-fidelity rendering region unchanged, or determines the position of the high-fidelity rendering region smoothly based on the historical gaze point trajectory using a prediction algorithm (such as "Kalman filter", linear extrapolation, etc.) to avoid visual flicker caused by frequent region jumps.

[0027] S3. Based on the content semantic category and the initial gaze area parameters, dynamically determine the geometric parameters of the high-fidelity rendering area (including center coordinates, radius, ellipticity or rectangular boundary). S4. Perform power consumption optimization operations on the display area outside the high-fidelity rendering area; Content recognition involves two main methods: firstly, using a locally deployed lightweight convolutional neural network (CNN) model to perform real-time semantic segmentation and classification of the current screen frame; secondly, or by calling the operating system's Accessibility API to directly obtain the UI element tree structure and attributes of the currently active window. Based on this information, the system determines the primary semantic category of the currently displayed content (e.g., dense text, wide-format image, video stream, or interactive UI).

[0028] The system automatically switches the gaze area mode based on the content type, focusing on text and reducing distractions while reading, and expanding the field of view to preserve details when viewing images. This is particularly useful on pages with mixed text and images. Figure 2 The text pattern shown (small circular gaze area) and Figure 3The image shows different rendering effects under the image mode (large elliptical gaze area). Among them, rendering operations that reduce power consumption include: for OLED or Micro LED screens, turning off the brightness of pixels outside the high-fidelity rendering area to pure black; for LCD screens, applying a filter that reduces visual sharpness outside the high-fidelity rendering area, such as Gaussian blur overlaid with a 50% grayscale mask.

[0029] S5. Monitor content changes within the power-optimized operation area. This is done by performing inter-frame difference operations, significant change detection algorithms, or listening to high-level events from the operating system (such as new notification pushes, window creation / destruction, and control state changes) to identify whether new visual elements or key content updates have appeared in the non-focused area. When a key change event matching a preset condition is detected, a temporary visual enhancement operation (such as brightening) is performed on the area where the key change event occurred. The duration of the visual enhancement operation is 300 milliseconds to 800 milliseconds.

[0030] Step S5, which involves monitoring changes in content, includes: The system identifies whether detected content changes exhibit periodicity. If periodicity is identified, temporary visual enhancement operations are suppressed. Specifically, time-series analysis is performed on the changing signal (e.g., applying Fast Fourier Transform or calculating the autocorrelation function) to determine if it possesses significant periodic characteristics (such as a fixed frame rate in video playback or a looping pattern in a GIF animation). If periodic activity is identified, temporary brightening is not triggered, but the content is allowed to remain displayed at the minimum brightness level necessary to maintain basic readability.

[0031] For non-periodic content changes, priority is assessed based on the semantic information of the changes, categorizing them into high-priority, medium-priority, and low-priority; only high-priority and medium-priority content changes are subject to temporary visual enhancement operations. Specifically: High priority: This includes messages pushed by the operating system notification center, application error pop-ups (e.g., pop-ups with 'Error' or 'Warning' in the window title bar), and new message notifications from instant messaging software. The system will trigger a strong brightness increase (e.g., increasing brightness to 100% and overlaying a dynamic pulse border). Figure 4 As shown, when critical information changes in areas outside the high-fidelity rendering area, the system will briefly and prominently highlight it to ensure the user doesn't miss it, then automatically return to energy-saving mode. The dynamic pulse border is a white ring with 100% brightness, flashing at a frequency of 2Hz for 500ms before fading out.

[0032] Medium priority: This includes email arrivals, social media updates, etc. The system will trigger standard brightening (e.g., moderately increasing brightness).

[0033] Low priority: This includes webpage ad carousels, irrelevant UI decorative refreshes, etc. The system will suppress any brightening operations to maintain the energy-saving state of this area.

[0034] This invention relates to an adaptive gaze point display system based on content awareness and user calibration, used to implement the aforementioned method, such as... Figure 5 As shown, the system includes: An eye-tracking module is used to acquire real-time user eye movement data and classify eye movement states using a preset velocity-acceleration threshold model or machine learning classifier, outputting the determination results of fixation events or saccade events. The eye-tracking module can be any device or software system capable of outputting the position information of the user's gaze point in the screen coordinate system. Its implementation forms include: 1. a hardware solution based on a dedicated infrared eye tracker; 2. a software solution based on a built-in ordinary camera (RGB or IR) combined with a lightweight neural network model; 3. a hybrid of solutions 1 and 2.

[0035] The content analysis module is used to identify the semantic category of the content currently displayed and to monitor content changes in non-focused areas; The decision engine receives the judgment results of gaze events and dynamically determines the geometric parameters of the high-fidelity rendering area based on the semantic category of the content and the user's individualized initial gaze area parameters. During saccade events, it outputs instructions to maintain the position of the high-fidelity rendering area unchanged, or to smoothly transition the position of the high-fidelity rendering area based on the historical gaze point trajectory through a prediction algorithm.

[0036] The partition rendering control module receives control commands from the decision engine, performs power optimization operations on non-high-fidelity rendering areas, and performs temporary visual enhancement operations on specified changing areas.

[0037] The decision engine integrates all input information, including: user calibration parameters, current content semantic category, eye movement event type (fixation / saccade), and the priority and pattern of change events.

[0038] In the state of a gaze event, the decision engine dynamically calculates and outputs the geometric parameters (center coordinates, radius, ellipticity, or rectangle) of the high-fidelity rendering area required for this refresh based on the scaling factor corresponding to the content category (e.g., text = 0.8, image = 1.2).

[0039] The decision engine generates temporary highlighting instructions, including the target area coordinates and highlighting intensity, only for high-priority or medium-priority changes that are determined to be non-periodic.

[0040] Finally, the rendering control module receives the above instructions, turns off the non-focus pixels on the OLED screen, and performs a brief (e.g., 500 milliseconds) brightening operation in the specified area.

[0041] This invention also supports user customization: The system provides an "Attention Range" configuration entry in the operating system's settings menu.

[0042] Users can choose the "Automatic Adaptation" mode (default), where the system takes over completely; or switch to the "Manual Setting" mode.

[0043] In manual mode, users can adjust the size of the attention range using intuitive slider controls and select from preset templates (such as "Programming", "Immersive Reading", and "Audio-Visual Entertainment") with a single click to quickly match different usage scenarios.

[0044] The system can dynamically adjust its strategy based on the device's current battery status: when the battery level is below 20%, it expands the off-field area and shortens the duration of the brightening effect; when the device is connected to a power source, it prioritizes ensuring information integrity.

[0045] As an alternative to the present invention, the following is provided: 1. Alternative Calibration Methods: In addition to active testing, passive calibration can be used. During natural user interaction, the system records areas frequently viewed (regression saccades) and automatically infers and expands the user's attention span.

[0046] 2. Content Recognition Alternative: If device computing power is limited, the content recognition logic can be simplified. For example, the application type can be determined only by the window title or process name (e.g., "VSCode" is considered code, "Photoshop" is considered an image), instead of analyzing every frame.

[0047] 3. Visual Enhancement Alternatives: In addition to visual highlighting, other sensory channels can be combined. For example, when a key change is detected, a slight tactile feedback (such as device vibration) or auditory cue (such as a short sound effect) can be triggered simultaneously to form a multimodal alert.

[0048] 4. Hardware platform replacement: The core logic of this solution can be migrated to AR / VR headsets to optimize the rendering pipeline and further save battery life of mobile devices.

[0049] 5. Eye-tracking event detection alternative: If the hardware sampling rate is insufficient, a simple velocity thresholding method based on the gaze point coordinates can be used (e.g., angular velocity > 30° / s is judged as saccade).

[0050] 6. Periodic Detection Alternative: For devices with limited computing resources, a whitelist of "known periodic sources" (such as specific video player windows or system tray areas) can be maintained to directly exempt them from change detection.

[0051] 7. Content Recognition Degradation Solution: On devices with extremely limited computing resources, the CNN model can be completely omitted, relying solely on window metadata provided by the operating system (such as process name and window title) for coarse-grained content classification. For example, windows from 'Code.exe' can be uniformly treated as 'text', and windows from 'Photoshop.exe' can be treated as 'images'.

[0052] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Unless otherwise specified, the embodiments and features described and illustrated in this application can be combined with each other. The components of the embodiments of this application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. An adaptive gaze point display method based on content awareness and user calibration, characterized in that: Includes the following steps: S1. When the function is first enabled, perform a visual attention range calibration test to obtain and store the user's individualized initial gaze area parameters; S2. The eye-tracking module acquires the user's gaze point position in real time and identifies the semantic category of the content displayed on the screen. S3. Dynamically determine the geometric parameters of the high-fidelity rendering region based on the content semantic category and the initial gaze area parameters; S4. Perform power consumption optimization operations on the display area outside the high-fidelity rendering area; S5. Monitor changes in content within the power consumption optimization operation area. When a key change event that meets the preset criteria is detected, perform a temporary visual enhancement operation on the area where the key change event occurred.

2. The adaptive gaze point display method based on content awareness and user calibration according to claim 1, characterized in that: The specific process of step S1 is as follows: The test screen is displayed, which includes a central gaze cue point and multiple interactive target elements randomly distributed in the area surrounding the central gaze cue point. Receive user clicks on interactive target elements via input devices and record the position coordinates of successfully clicked target elements; Based on the recorded location coordinates, the personalized attention range boundary centered on the central gaze cue point is calculated using a geometric fitting algorithm; The parameters of the personalized attention range boundary are stored as initial gaze area parameters.

3. The adaptive gaze point display method based on content awareness and user calibration according to claim 1, characterized in that: In step S2, the process of obtaining the user's gaze point position includes real-time classification of the user's eye movement data: Based on a preset velocity-acceleration threshold model or machine learning classifier, eye movement states are classified into fixation events or saccadic events.

4. The adaptive gaze point display method based on content awareness and user calibration according to claim 3, characterized in that: Step S3 is executed only when the current eye movement state is classified as a gaze event; when it is classified as a saccade event, the system maintains the position of the high-fidelity rendering area unchanged, or uses a prediction algorithm based on the historical gaze point trajectory to smoothly transition the position of the high-fidelity rendering area.

5. The adaptive gaze point display method based on content awareness and user calibration according to claim 1, characterized in that: Step S5, which involves monitoring changes in content, includes: Identify whether the detected content changes have periodic characteristics; If the content changes are identified as having periodic characteristics, then temporary visual enhancement operations are suppressed.

6. The adaptive gaze point display method based on content awareness and user calibration according to claim 5, characterized in that: For non-periodic content changes, priority is evaluated based on the semantic information of the content changes, and they are divided into high priority, medium priority and low priority. Perform temporary visual enhancements only on high-priority and medium-priority content changes.

7. The adaptive gaze point display method based on content awareness and user calibration according to claim 1, characterized in that: In step S4, performing a power-saving rendering operation includes: for OLED or Micro-LED screens, turning off the brightness of pixels outside the high-fidelity rendering area; for LCD screens, applying a filter that reduces visual clarity outside the high-fidelity rendering area.

8. The adaptive gaze point display method based on content awareness and user calibration according to claim 1, characterized in that: The duration of the visual enhancement operation is from 300 milliseconds to 800 milliseconds.

9. An adaptive gaze point display system based on content awareness and user calibration, used to implement the method as described in any one of claims 1 to 8, characterized in that: The system includes: The eye-tracking module is used to acquire the user's eye movement data in real time, and classify the eye movement state through a preset velocity-acceleration threshold model or machine learning classifier, and output the judgment result of fixation event or saccade event; The content analysis module is used to identify the semantic category of the content currently displayed and to monitor content changes in non-focused areas; The decision engine receives the judgment results of gaze events and dynamically determines the geometric parameters of the high-fidelity rendering area based on the semantic category of the content and the user's individualized initial gaze area parameters. During saccade events, it outputs instructions to maintain the position of the high-fidelity rendering area unchanged, or to smoothly transition the position of the high-fidelity rendering area based on the historical gaze point trajectory through a prediction algorithm. The partition rendering control module receives control commands from the decision engine, performs power optimization operations on non-high-fidelity rendering areas, and performs temporary visual enhancement operations on specified changing areas.

10. The adaptive gaze point display system based on content awareness and user calibration according to claim 1, characterized in that: The eye-tracking module includes a device or software system capable of outputting the position information of the user's gaze point in the screen coordinate system.

Images