View display method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a view display method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining drawing information after a view redrawing request sent by an application program is obtained, and obtaining at least one rendering layer according to the drawing information; when the time consumption of the rendering stage exceeds a first preset duration, aiming at any rendering layer, optimizing the rendering layer to obtain a first optimized layer, the data volume corresponding to the first optimized layer being smaller than the data volume corresponding to the rendering layer; wherein the rendering stage is used for indicating a stage from receiving a view redrawing request to obtaining at least one rendering layer; and performing synthesis operation on each first optimized image layer to obtain a first synthesis view, and displaying the first synthesis view on a screen, thereby avoiding performance bottleneck, improving synthesis speed in a synthesis stage, reducing a jamming phenomenon caused by excessive use of resources, and improving fluency of user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a view display method, device, electronic device, storage medium, and program product. Background Art

[0002] Stuttering refers to a brief pause or delay in the smoothness of video playback, game operation, or user interface interaction. To reduce stuttering, using the time to wait for vertical synchronization (VSYNC) and adding a buffer (triple buffering) are common techniques.

[0003] Triple buffering adds an extra buffer to the traditional double buffering mechanism, often called the standby buffer or third buffer. Under this mechanism, the graphics card can continue to render new frames in the standby buffer without having to wait for the VSYNC signal. When the display is ready to receive a new frame, the graphics card can immediately swap buffers and pass the contents of the standby buffer to the front buffer for display. This method reduces the idle time of the graphics card, improves rendering efficiency, and reduces the lag caused by waiting for VSYNC.

[0004] However, when processing highly complex images, if the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) resources are close to their computing capacity limits, it may still lead to performance bottlenecks, thereby causing lag. Summary of the Invention

[0005] The present application provides a view display method, device, electronic device, storage medium and program product to reduce the lag caused by excessive resource use and improve the smoothness of user experience.

[0006] In a first aspect, an embodiment of the present application provides a view display method, the method comprising:

[0007] After obtaining the view redraw request sent by the application, determining drawing information, and obtaining at least one rendering layer according to the drawing information;

[0008] When the duration of the rendering phase exceeds a first preset duration, for any rendering layer, the rendering layer is optimized to obtain a first optimized layer, where the amount of data corresponding to the first optimized layer is smaller than the amount of data corresponding to the rendering layer; wherein the rendering phase is used to indicate the phase from receiving the view redraw request to obtaining at least one rendering layer;

[0009] A synthesis operation is performed on each first optimized layer to obtain a first synthesized view, and the first synthesized view is displayed on a screen.

[0010] Optionally, the method further includes:

[0011] When the time consumed in the rendering phase does not exceed a first preset time length and the time consumed in the synthesis phase exceeds a second preset time length, optimizing any rendering layer to obtain a second optimized layer, wherein the amount of data corresponding to the second optimized layer is smaller than the amount of data corresponding to the rendering layer; performing a synthesis operation on each second optimized layer to obtain a second synthesized view, and displaying the second synthesized view on a screen;

[0012] The synthesis phase is used to indicate a phase of synthesizing each rendering layer. The synthesis phase begins with obtaining each rendering layer and ends with obtaining a second synthesized view.

[0013] Optionally, the first preset duration is less than or equal to a vertical synchronization VSYNC signal period.

[0014] Optionally, after obtaining the first optimized layer, the following steps are further included:

[0015] Storing each first optimized layer into a cache pool;

[0016] Accordingly, performing a synthesis operation on each first optimized layer to obtain a first synthesized view, and displaying the first synthesized view on a screen includes:

[0017] After the first VSYNC signal is triggered after each first optimized layer is stored in the buffer pool, each first optimized layer is taken out from the buffer pool, and a synthesis operation is performed on each first optimized layer to obtain a first synthetic view, and the first synthetic view is displayed on the screen.

[0018] Optionally, for any rendering layer, optimizing the rendering layer to obtain a first optimized layer includes:

[0019] Preprocessing the rendering layer to obtain a processing result, wherein the preprocessing includes: clipping processing, blur processing and highlight processing;

[0020] Determine the scenario type and select the target optimization model from multiple optimization models based on the scenario type;

[0021] Inputting the rendering layer, processing results and scene metadata into the target optimization model to obtain an optimization result;

[0022] The optimization result and the processing result are fused to obtain an optimized layer.

[0023] Optionally, the optimization result and the processing result are fused to obtain an optimized layer, including:

[0024] Determine scenario constraints;

[0025] Based on the potential interpolation technology, the optimization result and the processing result are fused to obtain the optimized layer, and during the fusion process, the scene constraint condition is injected through ControlNet.

[0026] In a second aspect, an embodiment of the present application provides a view display device, the device comprising:

[0027] A first obtaining module is used to determine drawing information after obtaining a view redrawing request sent by the application, and obtain at least one rendering layer according to the drawing information;

[0028] A second obtaining module is configured to, when the duration of the rendering phase exceeds a first preset duration, optimize any rendering layer to obtain a first optimized layer, wherein the amount of data corresponding to the first optimized layer is smaller than the amount of data corresponding to the rendering layer; wherein the rendering phase is used to indicate a phase from receiving the view redraw request to obtaining at least one rendering layer;

[0029] The display module is configured to perform a synthesis operation on each first optimized layer to obtain a first synthesized view, and display the first synthesized view on a screen.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0031] The memory stores computer-executable instructions;

[0032] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect above.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect above.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0035] The present application provides a view display method, device, electronic device, storage medium, and program product, the method comprising: after obtaining a view redrawing request sent by an application, determining drawing information, and obtaining at least one rendering layer based on the drawing information; when the time consumed in the rendering phase exceeds a first preset duration, optimizing any rendering layer to obtain a first optimized layer, the amount of data corresponding to the first optimized layer being less than the amount of data corresponding to the rendering layer; wherein the rendering phase is used to indicate the phase from receiving the view redrawing request to obtaining at least one rendering layer; performing a synthesis operation on each first optimized layer to obtain a first synthesized view, and displaying the first synthesized view on the screen, which can avoid performance bottlenecks, improve the synthesis speed of the synthesis phase, reduce the jamming phenomenon caused by excessive resource utilization, and improve the smoothness of the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] Figure 1 An application scenario diagram provided for an embodiment of the present application;

[0038] Figure 2 A flowchart of a view display method provided in an embodiment of the present application;

[0039] Figure 3 A system architecture diagram of a view display system provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of an optimized rendering layer provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a normal frame display provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of an abnormal frame display provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of another abnormal frame display situation provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of the workflow of an APP monitoring layer provided in an embodiment of the present application;

[0045] Figure 9 A schematic diagram of the workflow of the SF monitoring layer provided in an embodiment of the present application;

[0046] Figure 10A schematic structural diagram of a view display device provided in an embodiment of the present application;

[0047] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0048] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0050] Stuttering refers to a phenomenon in which the image or operation is not smooth, with brief pauses or delays during video playback, game operation, or user interface interaction. This phenomenon is often caused by insufficient system resources, synchronization issues, or other performance bottlenecks. To reduce stuttering, developers and hardware manufacturers have adopted various technical measures, among which the use of the waiting time for vertical synchronization (VSYNC) and the addition of a buffer (i.e., triple buffering) are common techniques.

[0051] In the traditional double-buffering mechanism, the system uses two buffers: a front buffer and a back buffer. The front buffer is used to display the current frame, while the back buffer is used to draw the next frame. However, this mechanism may cause the GPU to idle while waiting for the VSYNC signal, thereby reducing rendering efficiency. In this case, triple buffering optimizes performance by adding an additional buffer (commonly called a spare buffer or third buffer).

[0052] In the triple buffering mechanism, the GPU can continue to render new frames in the standby buffer without waiting for the VSYNC signal. This means that even if the front buffer is being displayed and the back buffer is waiting to be swapped, the GPU can still perform rendering operations in the standby buffer.

[0053] When the display is ready to receive a new frame, the graphics card can immediately swap buffers, passing the contents of the standby buffer to the front buffer for display. By increasing the standby buffer, triple buffering reduces the graphics card's idle time and improves rendering efficiency. The graphics card can utilize its computing power to a greater extent, reducing the lag caused by waiting for VSYNC. Triple buffering allows the system to increase the frame rate without affecting display synchronization, providing a smoother visual experience. This is especially important for rapidly changing scenes and applications with high frame rate requirements (such as games).

[0054] Although triple buffering can significantly improve rendering efficiency and smoothness, when processing highly complex images, if the CPU or GPU resources are close to their computing power limits, it may still cause performance bottlenecks, thereby causing lag.

[0055] In view of this, the present application provides a view display method, which can determine the drawing information after obtaining the view redrawing request sent by the application, and obtain at least one rendering layer based on the drawing information, and detect the time consumption of the rendering stage. When the time consumption of the rendering stage exceeds the first preset time length, each rendering layer is optimized separately to obtain at least one corresponding optimization layer, and the data volume corresponding to the optimization layer is smaller than the data volume of the corresponding rendering layer. Then, each first optimization layer is synthesized to obtain a first synthetic view, and the first synthetic view is displayed on the screen. When the rendering stage takes too long, it means that the complexity of the picture is high. Active optimization can avoid performance bottlenecks, improve the synthesis speed of the synthesis stage, reduce the jamming phenomenon caused by excessive resource utilization, and improve the smoothness of the user experience.

[0056] Figure 1 An application scenario diagram is provided for an embodiment of the present application. In a smartphone running the Android framework, when an application needs to update the user interface, it sends a view redrawing request to the Android framework. After receiving the request, the Android framework will determine the drawing information and generate at least one rendering layer based on this information. If the time spent in the rendering stage exceeds the first preset duration, each rendering layer will be optimized separately to generate at least one first optimized layer. The data volume of the first optimized layer is smaller than the data volume of the original rendering layer. The rendering stage refers to the entire process from receiving the view redrawing request to generating at least one rendering layer. Next, all the first optimized layers are synthesized to generate a first synthetic view, and then the first synthetic view is sent to the hardware layer, which is responsible for displaying it on the screen.

[0057] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0058] Figure 2 This is a flow chart of a view display method provided in an embodiment of the present application. The execution subject of this embodiment can be any device that can perform computing tasks, process information, generate graphics and present the results on the display screen. For example, it can be a smartphone, tablet computer, etc. This application uses a smartphone as the execution subject for specific description. Figure 2 As shown, an embodiment of the present application provides a view display method, which may include:

[0059] Step 201: After obtaining a view redrawing request sent by an application, determine drawing information, and obtain at least one rendering layer according to the drawing information.

[0060] The drawing information refers to a set of data and instructions for directing how to present graphic elements on the screen.

[0061] Such information includes but is not limited to the following:

[0062] 1. Position and size: Define the position and size of graphic elements on the screen.

[0063] 2. Color and style: Specify the color, fill style, border style, etc. of graphic elements.

[0064] 3. Image resources: If an image needs to be drawn, the drawing information will include the resource path or reference of the image.

[0065] 4. Fonts and text styles: If text rendering is involved, the drawing information will include font type, size, color, and other text formatting information.

[0066] 5. Hierarchy: In complex user interfaces, drawing information may include hierarchical relationships between elements to ensure the correct rendering order.

[0067] 6. Transformation information: including information on transformation operations such as rotation, scaling, and translation.

[0068] 7. Transparency and blending modes: Define how transparency is handled and how different layers blend together.

[0069] This application is described using a smartphone based on the Android framework as a specific embodiment, but is not limited to the Android framework. The specific implementation process of smartphones equipped with other operating systems has similar technical principles and steps.

[0070] Specifically, in the Android framework, when an application needs to update the UI (User Interface), it calls `invalidate()` to request a redraw of the current view. After receiving the view redraw request sent by the application, the smartphone equipped with the Android framework first determines the drawing information of the view. The method for determining the drawing information is as follows:

[0071] First, views need to be laid out and measured to determine their size and position. The `measure()` method measures the view's size, while the `layout()` method positions the view within its parent container. After layout and measurement are complete, the Android framework calls the view's `onDraw()` method at the appropriate time. The `onDraw()` method executes on the UI thread (main thread), where developers can use the Canvas object for drawing operations.

[0072] After the drawing information of the view is prepared, the smartphone passes the information to the rendering thread, and the rendering thread obtains at least one rendering layer according to the drawing information.

[0073] Step 202: When the rendering phase takes longer than a first preset duration, optimize any rendering layer to obtain a first optimized layer, where the amount of data corresponding to the first optimized layer is smaller than the amount of data corresponding to the rendering layer; wherein the rendering phase indicates the phase from receiving the view redrawing request to obtaining at least one rendering layer.

[0074] The rendering phase in this application begins with receiving a view redrawing request sent by an application and ends with obtaining at least one rendering layer.

[0075] The first preset duration can be set according to the actual application scenario, and this application does not limit this.

[0076] The time consumption of the rendering stage is monitored. When the time consumption of the rendering stage exceeds a first preset time length, any rendering layer is optimized to obtain a first optimized layer, wherein the data volume corresponding to the first optimized layer is smaller than the data volume corresponding to the rendering layer.

[0077] This application does not limit the method for optimizing rendering layers.

[0078] In an optional implementation, the geometry can be simplified, such as reducing the number of polygons, using low-polygon models instead of high-polygon models, or using level of detail (LOD) technology to dynamically adjust the complexity of the model according to the distance.

[0079] In another optional implementation, the texture resolution may be reduced, for example, by using a compressed texture format, or by merging multiple textures into a texture atlas to reduce texture switching.

[0080] Optionally, the first preset duration is less than or equal to a vertical synchronization VSYNC signal period.

[0081] In this way, by ensuring that the rendering time does not exceed the VSYNC cycle, frame loss or freeze caused by excessive rendering time can be avoided, and the stability and reliability of the system can be enhanced.

[0082] Optionally, for any rendering layer, optimizing the rendering layer to obtain a first optimized layer includes:

[0083] Preprocessing the rendering layer to obtain a processing result, wherein the preprocessing includes: clipping processing, blur processing and highlight processing;

[0084] Determine the scenario type and select the target optimization model from multiple optimization models based on the scenario type;

[0085] Inputting the rendering layer, processing results and scene metadata into the target optimization model to obtain an optimization result;

[0086] The optimization result and the processing result are fused to obtain an optimized layer.

[0087] Among them, cropping processing (Cropping): based on target detection (such as YOLO or Mask R-CNN) to locate key areas, crop non-focus areas (such as background redundant parts).

[0088] Blurring: Use Gaussian blur or bilateral filtering on low-priority areas (such as dynamic backgrounds) to reduce the amount of high-frequency information calculations.

[0089] Highlighting: Emphasize core interactive elements through edge detection (Canny / Sobel) combined with a region enhancement algorithm (CLAHE).

[0090] The scene type is used to describe and classify different categories or features of an image or rendered scene. For example, the scene type may include static, dynamic, high contrast, low light, etc. The scene type may be determined in different ways, which is not limited in this application.

[0091] In an optional implementation, computer vision technology can be used to analyze the content of the rendering layer to identify the main elements in the scene (such as buildings, natural landscapes, people, etc.), thereby determining the scene type.

[0092] In another optional implementation, a machine learning model (such as a deep learning network) can be trained to automatically classify scene types. The model can be trained based on a large amount of labeled data to identify different scene features.

[0093] Scene metadata is structured information about a scene. It can include descriptive, technical, and semantic information about the scene. For example, scene metadata can include lighting, motion vectors, and so on. Scene metadata can be obtained in a variety of ways, which are not limited in this application.

[0094] In an optional implementation, the scene metadata may be manually added by a designer or a user.

[0095] In another optional implementation, scene metadata may be extracted from the scene through image analysis and computer vision techniques.

[0096] Specifically, the smartphone performs cropping, blurring, and highlighting on the rendering layer in sequence to obtain a processing result, determines the scene type, and determines the target optimization model corresponding to the scene type according to the first mapping relationship table, wherein the first mapping relationship table is used to indicate the correspondence between multiple scene types and multiple optimization models, obtains scene metadata, and splices the rendering layer, processing results, and scene metadata into a multimodal input, uses ViT (Vision Transformer) or CLIP (Contrastive Language–Image Pretraining) to perform cross-modal feature alignment, generates a unified semantic representation, and inputs the generated unified semantic representation into the target optimization model to obtain an optimization result, fuses the optimization result and the processing result, and obtains an optimized layer.

[0097] The following uses a specific example to illustrate the correspondence between scenario types and optimization models.

[0098] When the scene type is high dynamic, the corresponding optimization model can be Diffusion Model (such as StableDiffusion XL), which generates a rendering layer with clearer details through iterative denoising, solving the edge jagged problem caused by traditional algorithms.

[0099] When the scenario type is real-time interaction, the corresponding optimization model can be a lightweight Transformer (such as EfficientViT, Efficient Vision Transformer, Efficient Visual Transformer), which dynamically allocates computing resources through the attention mechanism and prioritizes enhancing the user's gaze area.

[0100] In this way, through cropping, blurring and highlighting, unnecessary details and data volume can be reduced, thereby reducing computational complexity and improving rendering speed. Blurring and highlighting can enhance the visual layering and key point highlighting effects of the image, making the final image more attractive. By identifying the scene type, specific optimization strategies can be applied to ensure the best performance of different types of scenes. The optimized layers provide smoother visual effects, improving the overall user experience. By optimizing data volume and computing requirements, the load on the CPU and GPU can be effectively reduced, reducing power consumption.

[0101] Optionally, the optimization result and the processing result are fused to obtain an optimized layer, including:

[0102] Determine scenario constraints;

[0103] Based on the potential interpolation technology, the optimization result and the processing result are fused to obtain the optimized layer, and during the fusion process, the scene constraint condition is injected through ControlNet.

[0104] Scene constraints refer to restrictions or conditions that must be met in a specific scene or environment, and may include, for example, geometric constraints, motion constraints, and the like.

[0105] Among them, geometric constraints involve the shape, size and relative position of objects.

[0106] Motion constraints are restrictions on the movement of an object, such as speed, acceleration, and path. In robotics, the movement of a robotic arm may be restricted by joint angles and velocities.

[0107] Among them, the scene constraints can be determined manually or automatically generated by analyzing the physical environment of the scene, such as geographical location, climate conditions, time factors, etc., and this application does not limit this.

[0108] ControlNet is a technology that can introduce control signals into neural networks.

[0109] Specifically, a latent space is first defined. This high-dimensional vector space is capable of representing the various features of the generated content. Within this space, interpolation techniques are used to smoothly transition between different latent vectors. The optimization and processing results are represented as latent vectors, respectively. Interpolation is then performed between the two latent vectors. The interpolated latent vectors are then passed through a decoder (typically part of a neural network) to generate a fused result.

[0110] During the generation process, these constraints can be injected into the generation process by introducing ControlNet into the generation network. Through ControlNet, the generated fusion results not only conform to the interpolation results of the latent space, but also must satisfy the defined physical and motion constraints, which ensures that the generation results are reasonable in practical applications.

[0111] By introducing scene constraints into the fusion process, we ensure that the generated optimized layers meet actual requirements such as physical boundaries and motion trajectories, avoiding unrealistic or unreasonable situations. Furthermore, by utilizing latent interpolation technology, we can smoothly fuse the optimization and processing results in the latent space, capturing the complex characteristics of the data, generating high-quality layers, and improving visual effects.

[0112] Step 203: Perform a synthesis operation on each first optimized layer to obtain a first synthesized view, and display the first synthesized view on a screen.

[0113] Specifically, the smartphone performs a synthesis operation on each first optimized layer to obtain a first synthetic view, writes the first synthetic view into a display buffer, and displays the first synthetic view on the screen after the first VSYNC signal is triggered after the first synthetic view is written into the display buffer.

[0114] The present application provides a view display method, which can determine drawing information after obtaining a view redrawing request sent by an application, and obtain at least one rendering layer based on the drawing information; when the time consumed in the rendering stage exceeds a first preset time length, for any rendering layer, the rendering layer is optimized to obtain a first optimized layer, and the data amount corresponding to the first optimized layer is smaller than the data amount corresponding to the rendering layer; wherein, the rendering stage is used to indicate the stage from receiving the view redrawing request to obtaining at least one rendering layer; a synthesis operation is performed on each first optimized layer to obtain a first synthetic view, and the first synthetic view is displayed on the screen, which can avoid performance bottlenecks, improve the synthesis speed of the synthesis stage, reduce the jamming phenomenon caused by excessive resource utilization, and improve the smoothness of the user experience.

[0115] Optionally, after obtaining the first optimized layer, the view display method provided in the embodiment of the present application further includes:

[0116] Storing each first optimized layer into a cache pool;

[0117] Accordingly, performing a synthesis operation on each first optimized layer to obtain a first synthesized view, and displaying the first synthesized view on a screen includes:

[0118] After the first VSYNC signal is triggered after each first optimized layer is stored in the buffer pool, each first optimized layer is taken out from the buffer pool, and a synthesis operation is performed on each first optimized layer to obtain a first synthetic view, and the first synthetic view is displayed on the screen.

[0119] The cache pool is a memory structure used to temporarily store layer data for quick access.

[0120] Specifically, after obtaining all the first optimization layers, the smartphone stores each first optimization layer into a cache pool. After the first VSYNC signal is triggered after each first optimization layer is stored in the cache pool, the smartphone takes out each first optimization layer from the cache pool, and uses the GPU to perform a synthesis operation on the taken out first optimization layers to obtain a first synthetic view, and puts the first synthetic view into the view cache pool. After the first VSYNC signal is triggered after obtaining the first synthetic view, the smartphone takes out the first synthetic view from the view cache pool and displays the first synthetic view on the screen.

[0121] In the prior art, after obtaining at least one rendering layer, the at least one rendering layer is placed in a cache pool, and after the first VSYNC signal is triggered after the at least one rendering layer is placed in the cache pool, the at least one rendering layer is taken out and a compositing operation is performed. However, the present application does not place at least one rendering layer in the cache pool, but instead places first optimized layers after optimizing the rendering layer, and performs a compositing operation on each first optimized layer.

[0122] In this way, by placing the first optimized layer into the cache pool instead of the rendering layer, the time taken for subsequent synthesis stages can be reduced, synthesis efficiency can be improved, and the occurrence of lag can be reduced. Users will experience smoother interface interaction and faster response time when using the device, which is especially important for application scenarios that require high real-time and high performance.

[0123] Optionally, the view display method provided in the embodiment of the present application further includes:

[0124] When the time consumed in the rendering phase does not exceed a first preset time length and the time consumed in the synthesis phase exceeds a second preset time length, optimizing any rendering layer to obtain a second optimized layer, wherein the amount of data corresponding to the second optimized layer is smaller than the amount of data corresponding to the rendering layer; performing a synthesis operation on each second optimized layer to obtain a second synthesized view, and displaying the second synthesized view on a screen;

[0125] The synthesis phase is used to indicate a phase of synthesizing each rendering layer. The synthesis phase begins with obtaining each rendering layer and ends with obtaining a second synthesized view.

[0126] Among them, the synthesis stage starts with obtaining each rendering layer, and the timing starts from obtaining each rendering layer. After the first preset time, if the second synthetic view has not been obtained, that is, the synthesis stage has not ended, it is confirmed that the time consumed by the synthesis stage exceeds the second preset time. At this time, the synthesis stage is still continuing. The smartphone optimizes any rendering layer to obtain a second optimized layer. The amount of data corresponding to the second optimized layer is smaller than the amount of data corresponding to the rendering layer. Among them, the method of optimizing the rendering layer to obtain the second optimized layer refers to the above embodiment and will not be repeated here.

[0127] After obtaining each second optimized layer, each second optimized layer is placed in a cache pool. When the first VSYNC signal is triggered after each second optimized layer is placed in the cache pool, each second optimized layer is obtained from the cache pool, and a synthesis operation is performed on each second optimized layer to obtain a second synthetic view. The second synthetic view is placed in a view cache pool, and when the first VSYNC signal is triggered after the second synthetic view is placed in the view cache pool, the second synthetic view is displayed on the screen.

[0128] The display time corresponding to the third synthetic view obtained after the above synthesis stage is completed is later than the display time of the second synthetic view.

[0129] In this way, when the compositing phase takes longer than the second preset duration, the rendering layer is optimized to reduce the amount of data, thereby speeding up the compositing process. This method can effectively reduce the time consumed in the compositing phase, improve overall compositing efficiency, reduce lag, and enhance the user experience.

[0130] Figure 3 This is a system architecture diagram of a view display system provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the system includes: APP (Application) layer, SF layer (SurfaceFlinger layer), APP monitoring layer, SF monitoring layer, algorithm layer and hardware layer. The functions of each layer are as follows:

[0131] APP layer:

[0132] When the APP has a drawing action, the APP layer is used to determine the drawing information and send the drawing information to the APP rendering thread for rendering to obtain at least one rendering layer.

[0133] SF layer:

[0134] When VSYNC-SF is triggered, SF will query the BufferQueue (cache pool) to see if there is a new rendering layer. If so, it will take out the new rendering layer and perform synthesis through the GPU, and give the synthesized view to the hardware layer to refresh it on the screen.

[0135] APP monitoring layer

[0136] The APP monitoring layer is used to monitor the time consumed by the APP rendering phase and store each completed layer. When the next rendering times out, the previous layer will be rewritten to replace the current layer to ensure that the SF end can get a usable layer. When a timeout occurs in the rendering phase, the rendered layer will also be stored after the time is up, and this layer will be put into the algorithm layer for optimization.

[0137] SF monitoring layer

[0138] The SF monitoring layer monitors the duration of the SF synthesis phase. Its basic function is to store each completed view. When the next synthesis timeout occurs, the previous view is overwritten to replace the current synthesized view and sent to the GPU hardware layer for refresh. After the current synthesized view expires, it is stored for the next synthesis. If a synthesis timeout is detected, the rendering layer is optimized in the algorithm layer and then synthesized.

[0139] Algorithm layer:

[0140] The algorithm layer is used to optimize the rendering layer. Figure 4 This is a schematic diagram of an optimization of a rendering layer provided in an embodiment of the present application, such as Figure 4As shown, the incoming rendering layer is first preprocessed to obtain the processing result. The preprocessing includes clipping, blurring and highlighting. Then the scene type is determined, and the target optimization model is determined according to the scene type. The rendering layer, processing result and scene metadata are input into the target optimization model to obtain the optimization result. Based on the potential interpolation technology, the optimization result and the processing result are fused to obtain the fusion result. The fusion result and the scene constraint condition are input into the generation network to obtain the optimization layer, wherein the scene constraint condition is injected into the generation network through ControlNet. The preprocessing network can be trained by using reinforcement learning (PPO algorithm) to dynamically adjust the parameters of the preprocessing network (such as blur radius, clipping ratio, etc.). When performing quality assessment, objective indicators can be used. For example, PSNR / SSIM evaluates image fidelity and FPS evaluates rendering efficiency. Feedback training links can also be added to form a closed loop.

[0141] Among them, the feedback training link can include data collection and model updates. Data collection is to record the rendering layer data before and after optimization, user interaction behavior (such as click / gaze heat maps), hardware performance indicators (GPU / CPU load), etc. Model updates include offline training and online learning. Offline training uses contrastive learning (SimCLR framework) to explore the potential correlation between rendering layers before and after optimization. Online learning is to use LoRA or QLoRA to perform lightweight fine-tuning on the basic large model to adapt to specific device characteristics (such as mobile NPU acceleration).

[0142] The following is a detailed description of the frame loss. Figure 5 This is a schematic diagram of a normal frame display provided by an embodiment of the present application, as shown in FIG. Figure 5 As shown, the time it takes for the APP to render and the SF to synthesize a frame is all within one Vsync cycle (the Vsync cycle is related to the refresh rate of the current device, such as 8.33ms for a 120HZ Vsync cycle and 16.66ms for a 60HZ Vsync cycle).

[0143] Figure 6 A schematic diagram of an abnormal frame display situation provided in an embodiment of the present application is shown as follows: Figure 6 As shown in the figure, when the jam occurs, one situation is that the time taken for APP rendering exceeds one Vsync cycle, which causes SF to be unable to obtain the rendering layer from the cache pool after Vsync-sf arrives.

[0144] Figure 7 Another schematic diagram of abnormal frame display provided in the embodiment of the present application is as follows: Figure 7As shown in the figure, another situation when lag occurs is that the time taken to synthesize SF exceeds one Vsync cycle, resulting in SF being unable to respond to the current Vsync-sf signal in time. It can only obtain the view synthesized in the previous Vsync cycle when the next Vsync-sf signal arrives.

[0145] Figure 8 A schematic diagram of the workflow of an APP monitoring layer provided in an embodiment of the present application is shown as follows: Figure 8 As shown, when a normal rendering layer A is taken out by SF for synthesis, the rendering layer A will be stored in the App monitoring layer; a monitoring will be performed when a normal Vsync-APP arrives. When a freeze or frame loss occurs, and it is because the previous rendering stage timed out, the response of the APP monitoring layer will be triggered. At this time, the rendering layer A will be given to the cache pool. When Vsync-sf arrives, SF will normally take the rendering layer A for synthesis. After rendering B is completed, the algorithm layer of rendering layer B is optimized to obtain rendering layer C. The optimized rendering layer C is placed in the cache pool. When Vsync-sf arrives, SF will normally take the rendering layer C for synthesis.

[0146] Figure 9 A schematic diagram of the workflow of the SF monitoring layer provided in an embodiment of the present application is shown as follows: Figure 9 As shown, a monitor will be performed when a normal Vsync-SF arrives. When a freeze or frame loss occurs, and it is due to a timeout in the synthesis stage, the response of the SF monitoring layer will be triggered, and the latest rendering layer information will be put into the algorithm layer for optimization to obtain the rendering layer C. When Vsync-sf arrives, SF will normally take the rendering layer C for synthesis and refresh it to the screen through the hardware layer.

[0147] The view display method provided in the embodiment of the present application is based on the logic of display loading in one frame of Android. It does not distinguish between specific soc platforms and can be used on devices based on the Android system. It can also provide relevant visual interfaces in the settings. The user needs to be informed of the benefits and risks of turning on the execution logic in this embodiment. If the user has performance requirements, for example, if the smoothness of the whole machine is more important, it can be turned on to strip off the various functions. The division of labor of each layer is separate, and the interactive interface has only the algorithm layer, which can reduce the redundancy between the various functional layers and clarify the business functions of each functional layer. The present application can monitor the jams on multiple layers (APP rendering layer, SurfaceFlinger synthesis layer). When jams occur, the current rendering layer is optimized by one layer, and then optimized by another layer through the interactive algorithm layer. This ensures that the frame finally displayed to the user on the jammed frame can reduce the jam feeling.

[0148] Corresponding to the above view display method, the embodiment of the present application further provides a view display device, Figure 10 This is a structural diagram of a view display device provided in an embodiment of the present application. Figure 10 As shown, the device is applied to any device that can perform computing tasks, process information, generate graphics, and present the results on a display screen. For example, it can be a smart phone, tablet computer, etc. The device includes:

[0149] The first obtaining module 1001 is used to determine drawing information after obtaining a view redrawing request sent by the application, and obtain at least one rendering layer according to the drawing information;

[0150] The second obtaining module 1002 is configured to optimize any rendering layer to obtain a first optimized layer when the rendering phase lasts longer than a first preset duration, wherein the data volume corresponding to the first optimized layer is smaller than the data volume corresponding to the rendering layer; wherein the rendering phase is used to indicate a phase from receiving the view redraw request to obtaining at least one rendering layer;

[0151] The display module 1003 is configured to perform a synthesis operation on each first optimized layer to obtain a first synthesized view, and display the first synthesized view on a screen.

[0152] Optionally, the second obtaining module 1002 is further configured to:

[0153] When the time consumed in the rendering phase does not exceed a first preset time length and the time consumed in the synthesis phase exceeds a second preset time length, optimizing any rendering layer to obtain a second optimized layer, wherein the amount of data corresponding to the second optimized layer is smaller than the amount of data corresponding to the rendering layer; performing a synthesis operation on each second optimized layer to obtain a second synthesized view, and displaying the second synthesized view on a screen;

[0154] The synthesis phase is used to indicate a phase of synthesizing each rendering layer. The synthesis phase begins with obtaining each rendering layer and ends with obtaining a second synthesized view.

[0155] Optionally, the first preset duration is less than or equal to a vertical synchronization VSYNC signal period.

[0156] Optionally, the second obtaining module 1002 is further configured to:

[0157] Storing each first optimized layer into a cache pool;

[0158] Accordingly, the display module 1003 is specifically configured to:

[0159] After the first VSYNC signal is triggered after each first optimized layer is stored in the buffer pool, each first optimized layer is taken out from the buffer pool, and a synthesis operation is performed on each first optimized layer to obtain a first synthetic view, and the first synthetic view is displayed on the screen.

[0160] Optionally, when the second obtaining module 1002 optimizes any rendering layer to obtain a first optimized layer, it is specifically configured to:

[0161] Preprocessing the rendering layer to obtain a processing result, wherein the preprocessing includes: clipping processing, blur processing and highlight processing;

[0162] Determine the scenario type and select the target optimization model from multiple optimization models based on the scenario type;

[0163] Inputting the rendering layer, processing results and scene metadata into the target optimization model to obtain an optimization result;

[0164] The optimization result and the processing result are fused to obtain an optimized layer.

[0165] Optionally, when the second obtaining module 1002 fuses the optimization result and the processing result to obtain the optimized layer, it is specifically used to:

[0166] Determine scenario constraints;

[0167] Based on the potential interpolation technology, the optimization result and the processing result are fused to obtain the optimized layer, and during the fusion process, the scene constraint condition is injected through ControlNet.

[0168] The specific implementation principles and effects of the view display device provided in the embodiments of the present application can be found in the aforementioned embodiments and will not be repeated here.

[0169] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 11 As shown, the electronic device of this embodiment may include:

[0170] at least one processor 1101; and

[0171] a memory 1102 communicatively connected to the at least one processor;

[0172] The memory 1102 stores instructions that can be executed by the at least one processor 1101, and the instructions are executed by the at least one processor 1101 to enable the electronic device to execute the method as described in any of the above embodiments.

[0173] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .

[0174] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0175] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.

[0176] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the aforementioned embodiments when executed by a processor.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not implemented.

[0178] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0179] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include high-speed random access memory (Random Access Memory, RAM), and may also include non-volatile memory (NVM), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0180] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0181] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0182] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0183] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0185] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A view display method, characterized in that: The method comprises: After obtaining the view redraw request sent by the application, determining drawing information, and obtaining at least one rendering layer according to the drawing information; When the duration of the rendering phase exceeds a first preset duration, for any rendering layer, the rendering layer is optimized to obtain a first optimized layer, where the amount of data corresponding to the first optimized layer is smaller than the amount of data corresponding to the rendering layer; wherein the rendering phase is used to indicate the phase from receiving the view redraw request to obtaining at least one rendering layer; A synthesis operation is performed on each first optimized layer to obtain a first synthesized view, and the first synthesized view is displayed on a screen.

2. The method according to claim 1, characterized in that The method further comprises: When the time consumed in the rendering phase does not exceed a first preset time length and the time consumed in the synthesis phase exceeds a second preset time length, optimizing any rendering layer to obtain a second optimized layer, wherein the amount of data corresponding to the second optimized layer is smaller than the amount of data corresponding to the rendering layer; performing a synthesis operation on each second optimized layer to obtain a second synthesized view, and displaying the second synthesized view on a screen; The synthesis phase is used to indicate a phase of synthesizing each rendering layer. The synthesis phase begins with obtaining each rendering layer and ends with obtaining a second synthesized view.

3. The method according to claim 1, characterized in that The first preset duration is less than or equal to a vertical synchronization VSYNC signal period.

4. The method according to claim 1, wherein After obtaining the first optimized layer, the following steps are also included: Storing each first optimized layer into a cache pool; Accordingly, performing a synthesis operation on each first optimized layer to obtain a first synthesized view, and displaying the first synthesized view on a screen includes: After the first VSYNC signal is triggered after each first optimized layer is stored in the buffer pool, each first optimized layer is taken out from the buffer pool, and a synthesis operation is performed on each first optimized layer to obtain a first synthetic view, and the first synthetic view is displayed on the screen.

5. The method according to any one of claims 1 to 4, characterized in that For any rendering layer, optimizing the rendering layer to obtain a first optimized layer includes: Preprocessing the rendering layer to obtain a processing result, wherein the preprocessing includes: clipping processing, blur processing and highlight processing; Determine the scenario type and select the target optimization model from multiple optimization models based on the scenario type; Inputting the rendering layer, processing results and scene metadata into the target optimization model to obtain an optimization result; The optimization result and the processing result are fused to obtain an optimized layer.

6. The method according to claim 5, characterized in that The optimization result and the processing result are integrated to obtain an optimized layer, including: Determine scenario constraints; Based on the potential interpolation technology, the optimization result and the processing result are fused to obtain the optimized layer, and during the fusion process, the scene constraint condition is injected through ControlNet.

7. A view display device, characterized in that: The device comprises: A first obtaining module is used to determine drawing information after obtaining a view redrawing request sent by the application, and obtain at least one rendering layer according to the drawing information; A second obtaining module is configured to, when the duration of the rendering phase exceeds a first preset duration, optimize any rendering layer to obtain a first optimized layer, wherein the amount of data corresponding to the first optimized layer is smaller than the amount of data corresponding to the rendering layer; wherein the rendering phase is used to indicate a phase from receiving the view redraw request to obtaining at least one rendering layer; The display module is configured to perform a synthesis operation on each first optimized layer to obtain a first synthesized view, and display the first synthesized view on a screen.

8. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images