A rendering display method, device, equipment and readable storage medium

Abstract

The application discloses a rendering display method and device, equipment and a readable storage medium, and relates to the technical field of image processing. In the case that the current frame is output on the display interface, data is acquired from multiple dimensions to predict visual attention in the future, different regions of the display interface are determined by using the visual attention probability heat map, and the mixed weight corresponding to the pixel points is determined in combination with the different regions and the visual attention probability. The display content in the original frame to be displayed in the future and belonging to different regions is respectively subjected to hierarchical rendering according to different specified rendering methods. Based on the mixed weight, the rendering results are weighted and fused at the pixel level, and the target frame obtained through the fusion is output. The application has the technical effect of ensuring that the transition between different rendering layers is natural without abrupt visual boundaries in the case that there is no physical delay.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a rendering and display method, apparatus, device, and readable storage medium. Background Technology

[0002] As display devices achieve increasingly higher resolutions, the computational demands for rendering each frame increase exponentially, making it unsustainable to rely solely on boosting peak computing power. Foveated rendering technology can achieve high-quality rendering of the gaze area. However, from eye movement and image capture to high-quality rendering, the entire pipeline has an unavoidable physical delay (typically greater than 15 milliseconds). In fast-moving scenes, users perceive image lag or misalignment between the gaze point and the high-definition area.

[0003] Therefore, how to improve rendering technology to optimize user visual perception is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a rendering and display method, apparatus, device, and readable storage medium that uses multi-dimensional data to predict future visual attention and combines it with multi-region rendering. Finally, it performs adaptive blending based on pixel-level region weights, thereby ensuring a natural transition between different layers without abrupt visual boundaries without physical latency.

[0005] A rendering and display method, comprising: While the current frame is being displayed on the screen, acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data; Using the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface. Using the visual attention probability heatmap, different regions of the display interface are determined, and the mixed weights corresponding to the pixels are determined by combining the different regions and the visual attention probabilities in the visual attention probability heatmap. The display content belonging to different regions in the original frame that needs to be displayed in the future is rendered in layers according to different specified rendering methods; The multi-layer rendering results are weighted and fused at the pixel level using the hybrid weights to obtain the target frame, which is then output on the display interface.

[0006] Preferably, determining the hybrid weights corresponding to pixels by combining the different regions and the visual attention probabilities in the visual attention probability heatmap includes: Traverse the probabilities in the aforementioned visual attention probability heatmap; Calculate the region weight of the pixel corresponding to the current probability in different regions; The weights of the multiple regions are determined as the mixed weights.

[0007] Preferably, when the region includes a core region, a transition region, and an edge region, the regional weights of the pixel corresponding to the current probability in different regions are calculated, including: The current probability is smoothed by a step size adjustment to obtain the core region weight; Calculate the complementary value of the current probability and determine the complementary value as the edge region weight; The weights of the core region and the edge region are multiplied together, and the product of the multiplication result and a specified value is used as the weight of the transition region.

[0008] Preferably, the visual attention probability heatmap is used to determine different areas of the display interface, including: Obtain the prediction confidence of the visual attention probability heatmap; Acquire scene information and use the scene information to determine the scene movement speed; Obtain remaining power consumption budget data; The region threshold matching the current scene is dynamically determined using the prediction confidence level, the scene motion speed, and the remaining power budget data. The comparison result is obtained by comparing the region threshold with the visual attention probability in the visual attention probability heatmap; Using the comparison results, the display interface is divided into different areas.

[0009] Preferably, using the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface, including: The eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data are input into the fusion and prediction model; The fusion and prediction model divides the display interface into a grid, assigning the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data to the corresponding grid nodes; Based on the grid, spatial correlations are captured by graph convolution of the fusion and prediction model, and motion trends are learned through temporal convolution in the fusion and prediction model to obtain a visual attention probability heatmap corresponding to the screen resolution; wherein, the values ​​in the visual attention probability heatmap are the probability that a pixel region will become the center of gaze at a specified time point in the future.

[0010] Preferably, it further includes: When the target frame is output on the display interface, eye-tracking data, device operation data, interface semantic data, and historical behavior data are acquired. Using the newly acquired eye-tracking data, device operation data, interface semantic data, and historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface. The fusion and prediction model is tuned and trained using eye-tracking data corresponding to the target frame.

[0011] Preferably, the display content belonging to different regions in the original frame to be displayed in the future is rendered in layers according to different specified rendering methods, including: Asynchronously load Gaussian model parameters and texture data from system memory into cache or shared memory; For the core area display content, a micro neural network is used to color and mix Gaussian spheres in real time according to the viewpoint to generate a core area image with detailed and physically accurate blur after rendering; The rasterization pipeline is used to display the content of the transition area, reducing geometric complexity, and a simplified shader is used for shading to obtain the rendered image of the transition area. For the content displayed in the edge area, after reusing the image data from the previous frame, the resolution is updated or motion blur is applied to obtain the rendered edge area image. The core region image, the transition region image, and the edge region image are determined as the rendering result.

[0012] A rendering display device, comprising: The data acquisition module is used to acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data when the current frame is output on the display interface. The probability heatmap acquisition module is used to predict future visual attention using the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data, and to obtain a visual attention probability heatmap corresponding to the display interface. The region and weight determination module is used to determine different regions of the display interface using the visual attention probability heatmap, and to determine the mixed weights corresponding to the pixels by combining the different regions and the visual attention probabilities in the visual attention probability heatmap. The partitioned and layered rendering module is used to render the display content belonging to different regions in the original frame that will be displayed in the future in layers according to different specified rendering methods. The image fusion and display module is used to perform pixel-level weighted fusion of the multi-layer rendering results using the fusion weights to obtain the target frame and output the target frame on the display interface.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement any of the above-described rendering and display methods.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described rendering and display methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described rendering and display methods.

[0016] This application acquires data from four dimensions—eye tracking, device operation, interface semantics, and historical behavior—when the current frame is output on the display interface. Based on this data, it predicts future visual attention, obtaining a visual attention probability heatmap corresponding to the display interface. In other words, by comprehensively analyzing multi-dimensional data, it focuses on predicting future visual attention rather than current visual attention. This eliminates physical latency at the source, ensuring that the target frame after subsequent rendering will not experience delays. Then, to reduce rendering computational costs, the visual attention probability heatmap can be used to identify different regions of the display interface, and the hybrid weights corresponding to pixels can be determined by combining the visual attention probabilities of different regions with those in the heatmap. The display content belonging to different regions in the original frame to be displayed in the future is rendered in layers according to different specified rendering methods. That is, different regions use different rendering methods, allowing for high-standard rendering of highly focused areas and lower-standard rendering of less focused areas, thereby reducing computational costs and maintaining rendering efficiency for key areas. Finally, pixel-level weighted fusion of the multi-layer rendering results is performed using hybrid weights to obtain the target frame, which is then output to the display interface. Based on hybrid weights, pixel-level weighted fusion of the rendering results ensures natural transitions between different layers without abrupt visual boundaries.

[0017] In other words, this application has the technical effect of predicting future visual attention with multi-dimensional data, combining multi-region rendering, and finally performing adaptive mixing based on pixel-level region weights, thereby ensuring a natural transition between different layers without abrupt visual boundaries without physical delay. Attached Figure Description

[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating the implementation process of a rendering and display method provided in this application embodiment; Figure 2 This is a schematic diagram illustrating a specific process of a rendering and display method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a rendering display device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the specific structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] The following is a brief explanation of the technical terms used in this article.

[0023] Neural Gaussian Rendering: Neural Gaussian rendering is a groundbreaking technology that combines classic computer graphics with modern machine learning, capable of generating photorealistic 3D scene images in milliseconds.

[0024] Temporal reuse rendering: Temporal reuse rendering refers to the system reusing a large amount or even completely the image data calculated in the previous frame or even earlier when rendering the current frame, and only making minor updates where absolutely necessary, thereby minimizing rendering overhead.

[0025] Spatiotemporal Graph Neural Network (ST-GNN): A spatiotemporal graph neural network is a deep learning architecture designed to process data that simultaneously handles spatial relationships and temporal variations.

[0026] Graph Convolution: Graph convolutional networks are a type of neural network architecture that processes graph data by using the structural information of the graph to learn the representation of nodes.

[0027] Temporal convolution: Temporal convolutional networks are a type of model based on convolutional neural networks used for time series prediction and modeling.

[0028] Fallback mechanism: refers to providing a backup plan or error handling strategy when the system encounters an exception or error, in order to ensure the reliability and stability of the system.

[0029] Robustness: Robustness refers to the ability of a system, algorithm, or model to maintain stability and functional integrity when faced with abnormal inputs, disturbances, or uncertainties.

[0030] Ray tracing: Ray tracing is a rendering technique that uses algorithms to simulate the physical characteristics of light in the real world. It can achieve physically accurate shadows, reflections and refractions, as well as global illumination, making objects in virtual game scenes more realistic.

[0031] Mesh Shader: A new GPU programming model designed to address the bottlenecks of traditional graphics pipelines when handling complex geometry. Mesh Shader provides a more flexible and efficient way to process large amounts of geometry, especially in modern real-time 3D rendering environments.

[0032] Temporal supersampling: Temporal supersampling is a key graphics rendering optimization technique. It efficiently generates the current frame's image by reusing and recombining historical pixel information across time (frames). Its core goal is to achieve high image quality or high frame rate with extremely low computational overhead.

[0033] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Please refer to Figure 1 The rendering and display method includes the following steps.

[0035] S101. When the current frame is output on the display interface, acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data.

[0036] Among them, eye-tracking data is used to indicate where the user is currently looking, such as eye movement coordinate sequence, real-time pupil center coordinates, motion vector (velocity, acceleration), and blink status.

[0037] Device operation data, such as cursor position, movement trajectory, click events, and keyboard input events, can be used to assess current or future potential visual attention positions.

[0038] Interface semantic data refers to the specific content displayed on the current interface, such as the current application window type (game, document, browser), UI element layout, etc. The user's visual attention and changes vary depending on the displayed content.

[0039] Historical behavior data refers to data on a user's historical behavior, such as a user's personalized historical gaze patterns (e.g., the habitual scanning path when reading a document), or historical data on which a user will click on a certain part of an interface or which part they habitually pay attention to.

[0040] After acquiring eye-tracking data, device operation data, interface semantic data, and historical behavior data, all data can be time-aligned to facilitate subsequent operations. Then, data cleaning and normalization can also be performed.

[0041] S102. Using eye-tracking data, device operation data, interface semantic data, and historical behavior data, predict future visual attention and obtain a visual attention probability heatmap corresponding to the display interface.

[0042] After obtaining multi-dimensional visual attention-related data, future visual attention can be predicted based on this data, thereby obtaining a visual attention probability heatmap corresponding to the display interface.

[0043] This visual attention probability heatmap corresponds to the screen resolution, and the values ​​within it represent the probability that the pixel region will become the center of gaze at a specified point in the future.

[0044] Specifically, a multimodal fusion prediction engine can be designed. This engine doesn't concern itself with where the user is currently looking, but rather focuses on predicting which areas of the screen the user's visual attention is most likely to be distributed in approximately 1-3 frames ahead. Specifically, this engine can predict future visual attention based on eye-tracking data, device operation data, interface semantic data, and historical behavior data, thereby outputting a visual attention probability heatmap.

[0045] In one specific embodiment of this application, eye-tracking data, device operation data, interface semantic data, and historical behavior data are used to predict future visual attention, resulting in a visual attention probability heatmap corresponding to the display interface, including: Eye-tracking data, device operation data, interface semantic data, and historical behavior data are input into the fusion and prediction model; The fusion and prediction model divides the display interface into a grid, assigning eye-tracking data, device operation data, interface semantic data, and historical behavior data to the corresponding grid nodes. Based on the grid, spatial correlations are captured by graph convolution of the fusion and prediction model, and motion trends are learned by temporal convolution in the fusion and prediction model to obtain a visual attention probability heatmap corresponding to the screen resolution; where the values ​​in the visual attention probability heatmap are the probability that a pixel region will become the center of gaze at a specified time in the future.

[0046] Specifically, this fusion and prediction model can employ a lightweight spatiotemporal graph neural network (ST-GNN) as its core. The screen space is divided into a grid, with each grid node containing the aforementioned multimodal features. The network captures spatial relationships (such as cursor position often guiding the gaze point) through graph convolutions and learns motion trends through temporal convolutions. Finally, the output is a probability heatmap corresponding to the screen resolution, where the value represents the probability that the pixel region will become the gaze center at a specified time in the future.

[0047] Specifically, in this embodiment, the screen is transformed into a dynamic image to achieve pixel-level visual attention probability prediction. In traditional neural network image processing, the screen is typically treated as a regular pixel grid (i.e., a two-dimensional matrix), scanned using a sliding convolution kernel. However, this presents two challenges: First, elements on the screen (buttons, text, cursor) have semantic relationships, rather than simple pixel proximity relationships (for example, a distant "OK" button may be strongly correlated with a nearby "Cancel" button). Second, user attention is a spatiotemporal unity—it requires considering both spatial layout (where the focus is) and temporal evolution (how the gaze moves).

[0048] To address the aforementioned challenges, embodiments of this application abstract the screen as a graph, which includes nodes and edges.

[0049] In this context, a node is a grid (e.g., 16×16) that divides the screen into N×N grids, with each grid cell representing a node. Each node contains the multimodal features of that region.

[0050] Edges represent the connections between nodes. Typically, it is initially assumed that adjacent grids are connected (spatial proximity), and then more complex semantic connections (such as similar UI elements across regions) are automatically learned through graph convolution.

[0051] Regarding the construction of input features, each grid node v_ij has a feature vector at each time t, which fuses multiple signals, including the following features: Visual semantic features (from scene understanding): The current screen is analyzed in real time by a very lightweight image segmentation model (such as a MobileNet variant), and the semantic category probability of each grid is output: the probability that the grid is "text", "button", "image", "face" or "blank background".

[0052] For example, a grid that includes a "Submit" button will have a high probability of being classified as a "Button".

[0053] Interactive hotspot features include: whether it contains a cursor: whether there is a mouse cursor in the grid (0 / 1 flag); whether it contains an eye tracker: whether there is a real-time eye tracker in the grid (0 / 1 flag); gaze duration: the cumulative time the eye tracker stays in the grid (after normalization); cursor speed: the instantaneous speed of the cursor when it passes through the grid.

[0054] Low-level visual features include: color contrast: local contrast within the grid (high-contrast areas are more likely to attract attention); edge density: the intensity of the image gradient within the grid.

[0055] Ultimately, the feature vector of each node v_ij is a high-dimensional vector (e.g., 128-dimensional) containing all the information mentioned above. The entire screen then becomes a node feature matrix of size (N×N) × feature dimension.

[0056] For spatial association capture, the goal of the graph convolution within it is to teach each node to see its neighbors. For example, a "text" node needs to know that it is immediately below a "button" in order to infer that the user might switch from reading text to clicking the button.

[0057] Simplified mathematical process of graph convolution: For a center node v, its updated feature h_v' consists of two parts: text h_v' = σ( W_self h_v + W_neighbor (Aggregate neighbor features); h_v: The original characteristics of node v itself.

[0058] Aggregate neighbor features: Sum (or average) the features h_u of all neighboring nodes u using weighted methods. The weights can be automatically learned through an attention mechanism (i.e., Graph Attention Network GAT), allowing the network to focus on neighbors that are more important for prediction (e.g., neighbors in the grid where the cursor is located may be given higher weights).

[0059] W_self and W_neighbor: Learnable weight matrices.

[0060] σ: Activation function (such as ReLU).

[0061] By stacking 2-3 layers of graph convolution, the receptive field of each node gradually expands. The first layer learns to see its direct neighbors, the second layer can see the neighbors of its neighbors, and so on. Ultimately, a "text" node can integrate information from the entire "text paragraph" it is in and even the associated "button".

[0062] In the current scenario: the cursor guides the gaze point: if the "cursor feature" of the grid where the cursor is located is strong, the graph convolution will propagate this signal to the surrounding grid. The nodes of the surrounding grid will realize: "There is a cursor near me, and I may be gazed at soon."

[0063] Understanding UI layout: If a "button" node and its "description text" node are adjacent on the graph, graph convolution will strengthen the connection weights between them, allowing the model to understand that this is a functional component.

[0064] In the temporal trend learning process, graph convolution only processes the spatial information of a single frame. To predict the future, it is also necessary to understand the changing trends of the past few frames. That is, it is necessary to construct a time series: for each node v_ij, we cache its feature sequence of the past K frames (e.g., 8 frames, about 130ms): text [h_t, h_{t-1}, h_{t-2}, ..., h_{t-K+1}].

[0065] This is a matrix of shape (K, feature dimension).

[0066] Temporal convolution (TCN) or gated recurrent unit (GRU) processes this sequence using one-dimensional temporal convolution (or GRU) to capture dynamic patterns, as follows: Eye movement trajectory: For example, if the "eye movement features" within a node go from non-existent to present and then back to non-existent in the past few frames, temporal convolution can identify that this is a process of saccade and infer that the gaze may fall on subsequent nodes in this direction at the next moment.

[0067] Cursor trend: If the cursor moves from left to right in consecutive frames, temporal convolution learns this velocity vector and predicts the node the cursor might enter at the next moment.

[0068] Interface changes: If a dialog box suddenly pops up in a certain area (a sudden change in visual semantic features), temporal convolution can capture this sudden change and improve the prediction probability of that area.

[0069] Output: The temporal convolutional layer outputs a new feature vector for each node. This vector not only contains information from the current frame but also incorporates the change patterns over a period of time. This vector encodes the node's future state.

[0070] Spatiotemporal fusion and final prediction include: Spatial path: graph convolution processing of the current frame to obtain the "current spatial layout" encoding; Temporal path: temporal convolution processing of historical frames to obtain the "motion trend" encoding.

[0071] Fusion: Two codes are concatenated or added together and input into a fully connected layer. The final output is the probability that the node will become the gaze center at a certain time in the future (e.g., 50ms in the future).

[0072] By performing the same operation on all nodes, a visual attention probability heatmap for future frames is obtained.

[0073] Since the fusion and prediction model is a trainable and optimizable neural network model, it can be continuously optimized in practical applications to achieve increasingly accurate results with repeated use.

[0074] S103. Using the visual attention probability heatmap, determine different areas of the display interface, and combine the different areas with the visual attention probability in the visual attention probability heatmap to determine the hybrid weights corresponding to the pixels.

[0075] Since the visual attention probability heatmap contains the probability value of visual attention for each pixel region, the real interface can be divided into different regions with different levels of visual attention based on these probability values ​​to achieve layered rendering.

[0076] Specifically, in practical applications, the number of regions the display interface is divided into, and the probability intervals for each region, can be determined based on the actual situation. For instance, the display interface can be divided into two regions based on a single threshold: one with high visual attention and one with low visual attention. Alternatively, it can be divided into multiple regions of varying degrees, such as core regions, transition regions, and edge regions. More regions can also be divided based on finer-grained thresholds. Regions at the same level of visual attention can be a single area or multiple areas. That is, if an edge region is set, it can be a single, continuous area or multiple separate areas.

[0077] In practical applications, different thresholds can be preset to divide different areas, or the thresholds can be determined dynamically.

[0078] In one specific embodiment of this application, visual attention probability heatmaps are used to determine different areas of the display interface, including: Obtain the prediction confidence of the visual attention probability heatmap; Acquire scene information and use it to determine the scene's motion speed; Obtain remaining power consumption budget data; Dynamically determine the region threshold that matches the current scene using prediction confidence, scene motion speed, and remaining power budget data; The comparison results are obtained by comparing the region threshold with the visual attention probability in the visual attention probability heatmap. Using the comparison results, the display interface will be divided into different areas.

[0079] Specifically, a dynamic layered rendering decision-maker can be designed to determine different thresholds based on the specific circumstances of the current scene. The following example, using the division of the display interface into three regions with different levels of visual attention—core area, transition area, and edge area—illustrates how to determine the segmentation thresholds.

[0080] Core area (probability > threshold A): Marked as the area that must be rendered with the highest quality. Such as 1-2 visual focal points.

[0081] Transition zone (threshold B < probability < threshold A): The area along the path the eye may scan. A certain degree of visual continuity needs to be ensured.

[0082] Edge region (probability < threshold B): The area that will hardly be noticed in the present and in the short term.

[0083] In this embodiment, thresholds A and B are not fixed values, but are dynamically adjusted based on prediction confidence, scene motion speed, and remaining GPU power budget. For example, in game scenes with rapidly rotating camera angles, the transition area is expanded; in static document scenes, the core area is shrunk to maximize energy efficiency.

[0084] In practical applications, a baseline range can be set, within which the decision-maker dynamically shifts. Below are typical setup references and examples: Typical benchmark ranges are shown in the table below:

[0085] The region between two thresholds (with probabilities between B and A) is the transition zone, and visual continuity needs to be ensured. Examples for different scenarios are given below.

[0086] Scenario 1: Static office work (such as reading a document), state: the user is reading a long document, the eye movement is regular (scanning from left to right), the head is basically still, and the system power consumption budget is sufficient.

[0087] Dynamic adjustment: If the prediction confidence is high, the core area can be appropriately narrowed to concentrate computing power.

[0088] The motion in this scene is extremely low, therefore, the transition zone can be appropriately reduced.

[0089] The actual threshold can be set as follows: A = 0.85 (only areas with a probability higher than 85% enter the core area); B = 0.35 (only areas with a probability lower than 35% enter the edge area).

[0090] Result: Core area: A small area around the currently being read word is rendered using neural Gaussian rendering, resulting in extremely clear text edges.

[0091] Transition zone: The remaining text in the current line and the beginning of the next line are displayed in medium quality to ensure continuity during scanning.

[0092] Edge areas: around the screen, headers and footers, use time-domain multiplexing, which consumes almost no power.

[0093] Scenario 2: Intense gameplay (e.g., rapid head turning). Status: The user is rapidly turning their view in a shooting game, resulting in intense eye movements and high GPU load.

[0094] Dynamic adjustments: Low prediction confidence (due to fast motion) → The core area needs to be expanded to avoid sudden changes in image quality caused by prediction errors. Violent scene motion → The transition area needs to be expanded to provide a buffer for rapid scanning. Tight power budget → Edge areas can be appropriately compressed to allocate more areas for low-power rendering.

[0095] Actual thresholds: A = 0.70 (entering the core zone with a probability higher than 70%, which is more lenient than usual); B = 0.45 (entering the edge zone with a probability lower than 45%, which is stricter than usual, meaning more areas are classified as transition zones).

[0096] Result: Core area: A large region around the current gaze point (because of the fast movement, the gaze point may drift), which is rendered with high quality.

[0097] Transition Zone: A large area surrounding the core area, using optimized rasterization to ensure no screen tearing during camera rotation.

[0098] Edge region: Only the very small area at the very edge of the screen is used for temporal multiplexing.

[0099] Scenario 3: Laptop battery mode (e.g., power saving).

[0100] Status: The user is using the laptop to process emails and is not plugged in, and wants to extend battery life.

[0101] Dynamic adjustment: With extremely low power consumption budget, the decision-maker will systematically reduce the rendering quality of all areas while expanding the edge areas to save computing power.

[0102] Actual thresholds: A = 0.80 (maintain core area quality); B = 0.30 (only enter the edge area if the probability is less than 30%, which is stricter than usual and means that more areas are classified as edge areas).

[0103] Result: Core area: Only the body of the email currently being viewed is rendered using Neural Gaussian rendering.

[0104] Transition zone: Narrowed, most of the non-focused area is included in the edge zone.

[0105] Edge area: Significantly expanded, including inbox list, menu bar, etc., all of which are time-domain multiplexed, resulting in a significant reduction in GPU power consumption.

[0106] Regarding the threshold parameter baseline range setting and the distribution of the visual attention probability heatmap: In most scenarios, the visual attention probability heatmap is sparse, meaning that only 1-2 small areas have extremely high probabilities (>0.8), while large areas have extremely low probabilities (<0.3). Therefore, setting A to 0.7~0.9 can accurately lock the true visual focus, and setting B to 0.3~0.5 can cover most non-fixated areas.

[0107] Physiological characteristics of the human eye: Foveal vision covers only about 2 degrees of visual field, corresponding to a small area on the screen. Therefore, the core area does not need to be large, but the transition area needs to cover possible saccade paths.

[0108] System robustness: The threshold is dynamically adjustable, ensuring the system's adaptability in different scenarios and avoiding a decline in user experience due to prediction errors.

[0109] In summary, thresholds A and B are dynamic and adaptive, and their settings require a balance between image quality, performance, power consumption, and robustness. The numerical ranges mentioned above are empirical values ​​validated through extensive experimentation and can serve as a starting point for engineering implementation.

[0110] After identifying the different regions, and considering the need to merge the rendering results of each part later, in order to ensure a natural transition between layers without abrupt visual boundaries, pixel-level blending weights can be determined in this embodiment.

[0111] Specifically, the hybrid weights corresponding to pixels can be determined by combining the visual attention probabilities in different regions and the visual attention probability heatmap.

[0112] In other words, after the three-layer rendering is completed, the subsequent compositor does not simply stitch the images together. Instead, it reads the original visual attention probability heatmap and region segmentation, combining the two to determine the blending weights. This allows for pixel-level weighted fusion to generate the final frame for output to the display. This ensures an absolutely smooth visual transition. Specifically, for each pixel, different weights can be set for different regions; these weights are summed to 1, and these weights are the blending weights. The specific values ​​of the blending weights are related to the region and the visual attention probability in the visual attention probability heatmap.

[0113] In one specific embodiment of this application, determining the hybrid weight corresponding to a pixel by combining the visual attention probabilities in different regions and the visual attention probability heatmap includes: Traverse the probabilities in the visual attention probability heatmap; Calculate the region weight of the pixel corresponding to the current probability in different regions; The weights for multiple regions are determined as a mixed weight.

[0114] Specifically, when the region includes a core region, a transition region, and an edge region, the regional weight of the pixel corresponding to the current probability in different regions is calculated, including: The current probability is smoothed by a step size adjustment to obtain the core area weight; Calculate the complementary value of the current probability and determine the complementary value as the weight of the edge region; Multiply the weights of the core area and the edge area, and then multiply the result of the multiplication with a specified value twice to obtain the weight of the transition area.

[0115] Specifically, three weights can be calculated for each pixel: w_high (core region weight), w_mid (transition region weight), and w_low (edge ​​region weight), satisfying w_high + w_mid + w_low = 1. The weight allocation rules are as follows: Core region weight w_high: It approaches 1 when p is close to 1, and approaches 0 when p is close to 0. It is generated by the smoothing step function smoothstep(T_low,T_high,p), which smoothly transitions from 0 to 1 as p rises from T_low to T_high.

[0116] Edge region weight w_low: complementary to w_high, i.e., 1-w_high. It dominates when p is low and approaches 0 when p is high.

[0117] Transition region weight w_mid: To ensure a smooth transition between the three layers, intermediate weights are introduced. A common method is to construct this weight using the product of w_high and w_low, i.e., w_mid = 4. w_high w_low. This value reaches its maximum when p is in the middle region (such as around 0.5), and gradually decays to 0 towards both ends.

[0118] The three original weights were calculated and then normalized (divided by the sum of the three) to ensure that the sum was 1 and that each weight was a continuous value between 0 and 1.

[0119] S104. For the display content belonging to different areas in the original frame that needs to be displayed in the future, perform layered rendering according to different specified rendering methods.

[0120] After dividing the area into different regions, the content of the original frame that will be displayed in the future can be assigned to different regions and rendered using different specified rendering methods.

[0121] Specifically, for areas with high visual attention, a higher standard rendering method is used, while for areas with low visual attention, a lower standard rendering method is used.

[0122] The rendering methods for different regions can be specified in advance.

[0123] In one specific embodiment of this application, the display content belonging to different regions in the original frame to be displayed in the future is rendered in layers according to different specified rendering methods, including: Asynchronously load Gaussian model parameters and texture data from system memory into cache or shared memory; For the core area display content, a micro neural network is used to color and mix Gaussian spheres in real time according to the viewpoint to generate a core area image with detailed and physically accurate blur after rendering; The rasterization pipeline is used to display the content of the transition area, reducing geometric complexity, and a simplified shader is used for shading to obtain the rendered image of the transition area. For the content displayed in the edge area, after reusing the image data from the previous frame, the resolution is updated or motion blur is applied to obtain the rendered edge area image. The core area image, transition area image, and edge area image are determined as the rendering result.

[0124] Specifically, instruction distribution and parallel rendering can be performed: command packets are submitted to the GPU. The layered rendering executor drives multiple computing units of the GPU to work in parallel: Perform high-quality rendering (such as neural Gaussian rendering) on ​​the core area.

[0125] Perform optimized standard rendering on the transition area.

[0126] Perform extremely low-cost temporal multiplexing rendering on the edge regions.

[0127] At the same time, a separate asynchronous computing thread can be set up to preload the required data into the cache based on the predicted core region location of the next frame.

[0128] In other words, specifically, a high-quality rendering layer (for the core area) can be configured to use neural Gaussian rendering technology. This technology represents the scene as a series of 3D Gaussian spheres with learnable parameters (color, transparency, position, covariance). During rendering, a miniature neural network (microsecond-level inference) shades and blends these Gaussian spheres in real time according to the viewpoint, generating images with extremely detailed (such as complex lighting and fur texture) and physically accurate blur effects. Compared to traditional ray tracing, it achieves offline rendering quality while improving speed by 1-2 orders of magnitude.

[0129] Preloading mechanism: Based on the predicted core area location, the relevant Gaussian model parameters and texture data are asynchronously loaded from system memory into the GPU cache or shared memory in advance to eliminate access latency.

[0130] Standard and lightweight rendering layers (for transition and edge regions), where the transition region uses a rasterization pipeline optimized with a mesh shader to reduce geometric complexity and uses a simplified shader.

[0131] Edge region: Employs temporal oversampling technology to reuse image data from the previous frame, performing only very low-resolution updates or simple motion blur processing, significantly reducing GPU load.

[0132] S105. Use hybrid weights to perform pixel-level weighted fusion of the multi-layer rendering results to obtain the target frame and output the target frame on the display interface.

[0133] After rendering is complete, adaptive blending of the multi-layer rendering results can be performed. Specifically, for pixel composition of the target frame, its pixel colors can be obtained through weighted blending: text FinalColor=w_high×Color_High+w_mid×Color_Mid+w_low×Color_Low.

[0134] For example: When the probability is high (e.g., p>0.7): w_high is close to 1, and the image is mainly rendered with high quality in the core area. When the probability is low (e.g., p<0.3): w_low is close to 1, and the image is mainly rendered with light weight in the edge area. When the probability is in the middle: all three weights coexist, with the transition area having the highest weight, achieving a smooth blending of the core and edge areas and avoiding abrupt transitions.

[0135] This weighting based on continuous probability makes the transition of the image natural and smooth, ensuring the ultimate image quality in the gaze area and achieving the ultimate energy saving in the non-gaze area, while eliminating any possible visual abruptness.

[0136] In one specific embodiment of this application, it further includes: When the target frame is output on the display interface, acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data; By using newly acquired eye-tracking data, device operation data, interface semantic data, and historical behavior data, future visual attention is predicted, resulting in a visual attention probability heatmap corresponding to the display interface. The fusion and prediction model is tuned and trained using eye-tracking data corresponding to the target frame.

[0137] In other words, in this embodiment, eye-tracking data, device operation data, interface semantic data, and historical behavior data can be continuously acquired to continuously predict visual attention probability heatmaps. Based on the new visual attention probability heatmaps, the original frames to be displayed in the future are rendered, thereby continuously tracking and predicting the user's visual attention and prioritizing the allocation of limited computing power to the area of ​​user attention.

[0138] Furthermore, a continuous self-optimization loop can be built in. This involves data collection: recording the deviation data between the predicted heatmap and the actual gaze point; and online fine-tuning: using this deviation data, periodically (e.g., when the system is idle) the ST-GNN model in the multimodal fusion prediction engine is fine-tuned online in a learning-like manner, allowing the fusion and prediction model to continuously adapt to the user's personal habits and the specific environment of the device, achieving personalized optimization that becomes more accurate with use.

[0139] Applying the method provided in this application, when the current frame is output on the display interface, data is acquired from four dimensions: eye tracking, device operation, interface semantics, and historical behavior. Based on this data, future visual attention is predicted, resulting in a visual attention probability heatmap corresponding to the display interface. In other words, by comprehensively analyzing multi-dimensional data, the focus is not on current visual attention but on predicting future visual attention. This eliminates physical latency at the source, ensuring that the target frame after subsequent rendering will not experience delays. Then, to reduce the computational cost of rendering, the visual attention probability heatmap can be used to determine different regions of the display interface, and the hybrid weights corresponding to pixels are determined by combining the visual attention probabilities of different regions with those in the heatmap. The display content belonging to different regions in the original frame to be displayed in the future is rendered in layers according to different specified rendering methods. That is, different rendering methods are used for different regions, allowing for high-standard rendering of highly concerned areas and lower-standard rendering of less concerned areas, thereby reducing computational cost and maintaining rendering efficiency for key areas. Finally, pixel-level weighted fusion of the multi-layer rendering results is performed using hybrid weights to obtain the target frame, which is then output to the display interface. Based on hybrid weights, pixel-level weighted fusion of the rendering results ensures natural transitions between different layers without abrupt visual boundaries.

[0140] In other words, this application has the technical effect of predicting future visual attention with multi-dimensional data, combining multi-region rendering, and finally performing adaptive mixing based on pixel-level region weights, thereby ensuring a natural transition between different layers without abrupt visual boundaries without physical delay.

[0141] To facilitate a better understanding and implementation of the rendering and display method provided in the embodiments of this application by those skilled in the art, the rendering and display method will be described in detail below with reference to specific scenarios.

[0142] Please refer to Figure 2 The implementation process of this method is as follows: Phase 1: Perceiving Input and Prospective Prediction.

[0143] Event entry: The system continuously receives parallel data streams from multiple hardware devices, including eye-tracking coordinates, cursor vectors, touch events, and semantic events emitted by applications.

[0144] Fusion and Prediction: After all data is aligned temporally, it is fed into a multimodal fusion prediction engine. The engine doesn't care "where is the user now," but rather focuses on answering "in the next 1-3 frames, where is the user's visual attention most likely to be distributed on the screen?" It outputs a visual attention probability heatmap, which forms the basis for all subsequent decisions.

[0145] Phase Two: Real-time Decision Making and Resource Planning.

[0146] Dynamic Decision Making: The dynamic layered rendering decision maker receives a predictive heatmap. It combines the current system state (such as GPU load and remaining power budget) with scene dynamics (overall motion amplitude of the screen) to calculate a set of adaptive threshold parameters in real time.

[0147] Task planning: Based on these thresholds, the decision-maker transforms the continuous probability heatmap into discrete, executable instructions. It divides the screen space into a core region (must be of the highest quality), a transition region (must be continuous), and an edge region (can be significantly degraded), and packages this division with the corresponding rendering quality presets to generate a layered rendering command package.

[0148] Phase 3: Parallel execution and precise synthesis.

[0149] Instruction dispatch and parallel rendering: Command packets are submitted to the GPU. The layered rendering executor drives multiple computing units of the GPU to work in parallel: high-quality rendering (such as neural Gaussian rendering) is performed on the core region; optimized standard rendering is performed on the transition region; and low-cost temporal multiplexing rendering is performed on the edge region.

[0150] Meanwhile, a separate asynchronous computation thread preloads the necessary data into the cache based on the predicted core region location for the next frame.

[0151] Intelligent Compositing: After the three-layer rendering is completed, the compositor reads the original predicted probability heatmap as the blending weights, performs pixel-level weighted fusion, and generates the final frame output to the display. This ensures an absolutely smooth visual transition.

[0152] Phase Four: Feedback Collection and Self-Optimization.

[0153] Performance verification: Once the frame is displayed on the screen, the closed-loop optimizer starts in the background. It collects the user's actual gaze data during the frame display and compares it with the previously corresponding predicted heatmap to calculate the prediction bias.

[0154] Model Evolution: This bias data is safely cached. When the system is idle, the optimizer uses this data to perform lightweight incremental training on the AI ​​model in the prediction engine, updating its personalized parameters. This allows the system to make more accurate predictions the next time it encounters a similar interaction pattern.

[0155] In practical applications, this method involves four modules: a multimodal fusion prediction engine, a dynamic hierarchical rendering decision-maker, a hierarchical rendering executor, and a closed-loop optimizer. The functions of each module can be specifically referred to the method steps described in the above method embodiments, and will not be repeated here. The heatmaps mentioned in this article all refer to visual attention probability heatmaps.

[0156] This application achieves the following technical effects: significantly reduced perceptual latency: through pre-computation and pre-loading, screen response latency is significantly reduced, making interaction more responsive; significantly improved energy efficiency: by avoiding invalid rendering of areas not viewed by the user, the power consumption of the GPU in most scenarios can be significantly reduced, significantly improving the battery life of mobile devices; fundamentally improved visual experience: under the premise of unchanged hardware computing power, it provides ultra-high image quality for the core gaze area, while ensuring smooth and natural visual transitions, providing an unprecedented sense of immersion and clarity.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0158] The embodiments of this application also provide a rendering and display device, which can be referred to in conjunction with the rendering and display method described in the above method embodiments, and will not be described again here.

[0159] Please refer to Figure 3 The device includes: The data acquisition module 101 is used to acquire eye-tracking data, device operation data, interface semantic data and historical behavior data when the current frame is output on the display interface; The probability heatmap acquisition module 102 is used to predict future visual attention by utilizing eye-tracking data, device operation data, interface semantic data and historical behavior data, and obtain a visual attention probability heatmap corresponding to the display interface. The region and weight determination module 103 is used to determine different regions of the display interface using a visual attention probability heatmap, and to determine the mixed weights corresponding to pixels by combining the different regions and the visual attention probabilities in the visual attention probability heatmap. The partitioned and layered rendering module 104 is used to perform layered rendering of the display content belonging to different regions in the original frame that needs to be displayed in the future, according to different specified rendering methods. The image fusion and display module 105 is used to perform pixel-level weighted fusion of multi-layer rendering results using hybrid weights to obtain the target frame and output the target frame on the display interface.

[0160] With the current frame output on the display interface, data is acquired from four dimensions: eye tracking, device operation, interface semantics, and historical behavior. Based on this data, future visual attention is predicted, resulting in a visual attention probability heatmap corresponding to the display interface. In other words, by comprehensively analyzing multi-dimensional data, the focus is not on current visual attention but on predicting future visual attention. This eliminates physical latency at the source, ensuring that the target frame after subsequent rendering is latency-free. Then, to reduce rendering computational costs, the visual attention probability heatmap is used to identify different regions of the display interface, and the hybrid weights corresponding to pixels are determined by combining the visual attention probabilities of different regions with those in the heatmap. The content belonging to different regions in the original frame to be displayed in the future is rendered in layers using different specified rendering methods. That is, different regions use different rendering methods, allowing for high-standard rendering of highly focused areas and lower-standard rendering of less focused areas, thus reducing computational costs while maintaining rendering efficiency for key areas. Finally, pixel-level weighted fusion of the multi-layer rendering results is performed using hybrid weights to obtain the target frame, which is then output to the display interface. Based on hybrid weights, pixel-level weighted fusion of the rendering results ensures natural transitions between different layers without abrupt visual boundaries.

[0161] In other words, this application has the technical effect of predicting future visual attention with multi-dimensional data, combining multi-region rendering, and finally performing adaptive mixing based on pixel-level region weights, thereby ensuring a natural transition between different layers without abrupt visual boundaries without physical delay.

[0162] In one specific embodiment of this application, the region and weight determination module is specifically used to traverse the probabilities in the visual attention probability heatmap; calculate the region weight of the pixel corresponding to the current probability in different regions; and determine the multiple region weights as a mixed weight.

[0163] In one specific embodiment of this application, when the region includes a core region, a transition region, and an edge region, the region and weight determination module is specifically used to perform smoothing step size processing on the current probability to obtain the core region weight; calculate the complementary value of the current probability and determine the complementary value as the edge region weight; multiply the core region weight and the edge region weight, and use the quadratic product of the multiplication result and the specified value as the transition region weight.

[0164] In one specific embodiment of this application, the region and weight determination module is specifically used to obtain the prediction confidence of the visual attention probability heatmap; obtain scene information and use the scene information to determine the scene motion speed; obtain remaining power budget data; dynamically determine the region threshold matching the current scene using the prediction confidence, scene motion speed and remaining power budget data; compare the region threshold with the visual attention probability in the visual attention probability heatmap to obtain the comparison result; and use the comparison result to divide the display interface into different regions.

[0165] In one specific embodiment of this application, the probability heatmap acquisition module is specifically used to input eye-tracking data, device operation data, interface semantic data, and historical behavior data into a fusion and prediction model; the fusion and prediction model divides the display interface into a grid, and assigns the eye-tracking data, device operation data, interface semantic data, and historical behavior data to corresponding grid nodes; based on the grid, the fusion and prediction model uses graph convolution to capture spatial correlations, and learns motion trends through temporal convolution in the fusion and prediction model to obtain a visual attention probability heatmap corresponding to the screen resolution; wherein, the values ​​in the visual attention probability heatmap represent the probability that a pixel region will become the center of gaze at a specified time point in the future.

[0166] In one specific embodiment of this application, the data acquisition module is used to acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data when the target frame is output on the display interface. The probability heatmap acquisition module is used to predict future visual attention by utilizing newly acquired eye-tracking data, device operation data, interface semantic data, and historical behavior data, and obtain a visual attention probability heatmap corresponding to the display interface. The model optimization module is used to fine-tune and train the fusion and prediction model using eye-tracking data corresponding to the target frame.

[0167] In one specific embodiment of this application, the partitioned and layered rendering module is specifically used to asynchronously load Gaussian model parameters and texture data from system memory into cache or shared memory; for the display content of the core area, a micro neural network is used to shade and mix Gaussian spheres in real time according to the viewpoint to generate a rendered core area image with detailed and physically accurate blur; for the display content of the transition area, the rasterization pipeline is used to reduce geometric complexity and shade using a simplified shader to obtain a rendered transition area image; for the display content of the edge area, the image data of the previous frame is reused, and resolution is updated or motion blur is applied to obtain a rendered edge area image; the core area image, transition area image, and edge area image are determined as the rendering result.

[0168] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the rendering and display method described above can be referred to each other.

[0169] See Figure 4 As shown, the electronic device includes: Memory 332 is used to store computer programs; The processor 322 is used to implement the steps of the rendering and display method of the above method embodiments when executing a computer program.

[0170] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0171] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0172] The steps in the rendering and display method described above can be implemented by the structure of an electronic device.

[0173] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below corresponds to the rendering and display method described above. This application also provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above rendering and display method embodiments at runtime.

[0174] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0175] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described rendering and display method embodiments.

[0176] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described rendering and display method embodiments.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A rendering and display method, characterized in that, include: While the current frame is being displayed on the screen, acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data; Using the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface. Using the visual attention probability heatmap, different regions of the display interface are determined, and the mixed weights corresponding to the pixels are determined by combining the different regions and the visual attention probabilities in the visual attention probability heatmap. The display content belonging to different regions in the original frame that needs to be displayed in the future is rendered in layers according to different specified rendering methods; The multi-layer rendering results are weighted and fused at the pixel level using the hybrid weights to obtain the target frame, which is then output on the display interface.

2. The method according to claim 1, characterized in that, Determining the hybrid weights corresponding to pixels by combining the visual attention probabilities in the different regions and the visual attention probability heatmap includes: Traverse the probabilities in the aforementioned visual attention probability heatmap; Calculate the region weight of the pixel corresponding to the current probability in different regions; The weights of the multiple regions are determined as the mixed weights.

3. The method according to claim 2, characterized in that, When the region includes a core region, a transition region, and an edge region, the region weights of the pixels corresponding to the current probability in different regions are calculated, including: The current probability is smoothed by a step size adjustment to obtain the core region weight; Calculate the complementary value of the current probability and determine the complementary value as the edge region weight; The weights of the core region and the edge region are multiplied together, and the product of the multiplication result and a specified value is used as the weight of the transition region.

4. The method according to claim 1, characterized in that, Using the visual attention probability heatmap, different regions of the display interface are determined, including: Obtain the prediction confidence of the visual attention probability heatmap; Acquire scene information and use the scene information to determine the scene movement speed; Obtain remaining power consumption budget data; The region threshold matching the current scene is dynamically determined using the prediction confidence level, the scene motion speed, and the remaining power budget data. The comparison result is obtained by comparing the region threshold with the visual attention probability in the visual attention probability heatmap; Using the comparison results, the display interface is divided into different areas.

5. The method according to claim 1, characterized in that, Using the eye-tracking data, device operation data, interface semantic data, and historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface, including: The eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data are input into the fusion and prediction model; The fusion and prediction model divides the display interface into a grid, assigning the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data to corresponding grid nodes; Based on the grid, spatial correlations are captured by graph convolution of the fusion and prediction model, and motion trends are learned through temporal convolution in the fusion and prediction model to obtain a visual attention probability heatmap corresponding to the screen resolution; wherein, the values ​​in the visual attention probability heatmap are the probability that a pixel region will become the center of gaze at a specified time point in the future.

6. The method according to claim 5, characterized in that, Also includes: When the target frame is output on the display interface, eye-tracking data, device operation data, interface semantic data, and historical behavior data are acquired. Using the newly acquired eye-tracking data, device operation data, interface semantic data, and historical behavior data, future visual attention is predicted to obtain a visual attention probability heatmap corresponding to the display interface. The fusion and prediction model is tuned and trained using eye-tracking data corresponding to the target frame.

7. The method according to any one of claims 1 to 6, characterized in that, For the display content belonging to different regions in the original frame that will be displayed in the future, layered rendering is performed according to different specified rendering methods, including: Asynchronously load Gaussian model parameters and texture data from system memory into cache or shared memory; For the core area display content, a micro neural network is used to color and mix Gaussian spheres in real time according to the viewpoint to generate a core area image with detailed and physically accurate blur after rendering; The rasterization pipeline is used to display the content of the transition area, reducing geometric complexity, and a simplified shader is used for shading to obtain the rendered image of the transition area. For the content displayed in the edge area, after reusing the image data from the previous frame, the resolution is updated or motion blur is applied to obtain the rendered edge area image. The core region image, the transition region image, and the edge region image are determined as the rendering result.

8. A rendering display device, characterized in that, include: The data acquisition module is used to acquire eye-tracking data, device operation data, interface semantic data, and historical behavior data when the current frame is output on the display interface. The probability heatmap acquisition module is used to predict future visual attention using the eye-tracking data, the device operation data, the interface semantic data, and the historical behavior data, and to obtain a visual attention probability heatmap corresponding to the display interface. The region and weight determination module is used to determine different regions of the display interface using the visual attention probability heatmap, and to determine the mixed weights corresponding to the pixels by combining the different regions and the visual attention probabilities in the visual attention probability heatmap. The partitioned and layered rendering module is used to render the display content belonging to different regions in the original frame that will be displayed in the future in layers according to different specified rendering methods. The image fusion and display module is used to perform pixel-level weighted fusion of the multi-layer rendering results using the fusion weights to obtain the target frame and output the target frame on the display interface.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the rendering and display method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the rendering and display method as described in any one of claims 1 to 7.

Images