Picture generation method and device and electronic equipment

Abstract

The invention discloses a picture generation method and device, and belongs to the technical field of image processing. The method comprises the steps that content to be displayed is converted into drawing description information; the drawing description information is structured data used for describing element attributes of elements in the to-be-displayed content; and generating a picture according to the drawing description information.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to a method, apparatus, and electronic device for generating a screen. Background Art

[0002] Currently, in the screen generation solution, each application directly uses the graphics acceleration hardware to render content in an independent process, generates an image frame buffer, that is, a 2D image, and hands it over to the display service process to synthesize the final screen.

[0003] However, with the increase in the number of applications, the number of image frame buffers has increased exponentially, occupying a large amount of graphics memory.

[0004] Therefore, in the process of screen generation, there is a problem of large graphics memory overhead. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide a method and apparatus for generating a screen, which can solve the problem of large graphics memory overhead.

[0006] In a first aspect, the embodiments of this application provide a method for generating a screen, which includes:[[]]

[0007] Converting the content to be displayed into drawing description information; the drawing description information is structured data for describing the element attributes of the elements in the content to be displayed;

[0008] Generating a screen according to the drawing description information.

[0009] In a second aspect, the embodiments of this application provide a device for generating a screen, which includes:[[]]

[0010] A conversion module, configured to convert the content to be displayed into drawing description information; the drawing description information is structured data for describing the element attributes of the elements in the content to be displayed;

[0011] A generation module, configured to generate a screen according to the drawing description information.

[0012] In a third aspect, the embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, the embodiments of this application provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] Fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run programs or instructions to implement the method described in the first aspect.

[0015] Sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0016] In an embodiment of the present application, by converting the content to be displayed into drawing description information, where the drawing description information is structured data for describing the element attributes of the elements in the content to be displayed, and using the drawing description information to replace the image frame buffer, the memory occupancy of the drawing description information is much smaller than that of the image frame buffer, which can reduce the memory occupancy. According to the drawing description information, a picture is generated, which can reduce the graphics memory overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a method for generating a picture provided by an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a picture generation system provided by an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of another picture generation system provided by an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a redrawing area provided by an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of another redrawing area provided by an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of yet another picture generation system provided by an embodiment of the present application;

[0023] Figure 7 is a structural diagram of a picture generation device provided by an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0025] Figure 9 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The technical solutions of the embodiments of the present application will be clearly described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application belong to the scope of protection of the present application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0028] The screen generation method provided by the embodiments of the present application can be applied to at least the following application scenarios, which will be described below.

[0029] Currently, the working process of existing graphics systems includes: each application directly uses graphics acceleration hardware in its own process to render its own content. The result of the rendering is an image frame buffer, that is, a 2D image, which contains all the pixel data within the application window. After the rendering is completed, the application hands the image frame buffer to the display service process. The display service process is responsible for synthesizing the image frame buffers of multiple applications into the final screen and displaying it on the screen.

[0030] On the one hand, since each application renders independently in its own process and is affected by system scheduling and the application's own business logic, the animation effect calculation time and rendering completion time of different applications may be inconsistent. This will lead to an uncoordinated overall animation effect of the system. For example, the animation of one application may be faster or slower than that of other applications, resulting in visual inconsistency.

[0031] Exemplarily, the interface of the chat application is divided into two main parts:

[0032] Chat record area: Displays the message records between users. Input box area: The area where users input messages, which may include animation effects, such as cursor blinking, input prompt animation, etc.

[0033] When a new message arrives, the application needs to calculate the layout of the message, such as text wrapping, timestamp position, etc., and trigger the scrolling animation to display the latest message. The input box may need to process user input in real time, such as cursor movement, automatic completion prompts, and trigger corresponding animations, such as cursor blinking and prompt box pop-up.

[0034] Due to the influence of system scheduling and business logic, the calculation of the animation in the chat history area may be more time-consuming than that in the input box area, resulting in the animation of the two being out of sync. Rendering a large number of message records, especially those containing complex content such as pictures and emoticons, may take a long time. Rendering input boxes and their animations are usually lightweight and render quickly. Due to the difference in rendering completion time, the animation of the input box may have been completed and displayed, while the animation of the chat history area is still in progress, resulting in inconsistent interface display.

[0035] Users may notice that the cursor flashing and prompt animation of the input box has been completed, while the scrolling animation of the chat history area is still in progress, giving users a feeling that the interface is "stuck" or "unsmooth". The asynchronous animation will make the interface look inconsistent and reduce the user experience.

[0036] On the other hand, the rendering result of each application is an image frame buffer, that is, a 2D image. As the number of running applications increases, the number of image frame buffers will also increase exponentially. This will take up a lot of graphics memory, resulting in a waste of system resources.

[0037] On the other hand, when the display service process synthesizes the image frame buffers of multiple applications, it can only decide how to synthesize the picture based on the occlusion relationship of the layers. However, there is no sharing of content information between applications, so even if the content of an application is blocked by the windows of other applications, it will continue to render the blocked area. This will lead to invalid drawing operations, that is, the application draws content that will not actually be seen by the user, wasting computing resources and power consumption.

[0038] For example, two applications are open on the screen at the same time:

[0039] Application A: A browser window showing a web page.

[0040] Application B: A video player window, playing a video.

[0041] The window of application B is located above the window of application A, blocking part of the content of application A. The working method of the existing graphics system is: application A renders the content of the entire browser window, including the part blocked by application B. Application B renders the content of the entire video player window. After rendering, the two applications submit their respective image frame buffers to the display service process.

[0042] The display service process synthesizes two image frame buffers into a final picture according to the occlusion relationship of the layers. Although the window of Application B occludes a part of the content of Application A, Application A does not know that it is occluded. Therefore, Application A will still render the occluded area, even though this content will not be finally displayed on the screen.

[0043] Application A is still rendering the occluded rotating icon, consuming CPU and GPU resources, but these rendering results are not actually seen by the user. This ineffective rendering will lead to a waste of system resources. Especially when running multiple applications, the accumulation of ineffective rendering will significantly increase the system load. For mobile devices, ineffective rendering will increase power consumption and shorten the battery life.

[0044] In view of the problems in the related art, the embodiments of the present application provide a picture generation method and device, which can solve the problem of large graphics memory overhead in the related art.

[0045] The following will specifically describe the picture generation method provided by the embodiments of the present application with reference to the drawings and through specific embodiments and their application scenarios.

[0046] Figure 1 It is a flowchart of a picture generation method provided by the embodiments of the present application.

[0047] As Figure 1 shown, the picture generation method may include Step 110 - Step 120. This method is applied to a picture generation device and is specifically as follows:

[0048] Step 110, convert the content to be displayed into drawing description information; the drawing description information is structured data used to describe the element attributes of the content to be displayed;

[0049] Drawing description information: Structured data used to describe the attributes of elements in the content to be displayed. Converting the content to be displayed into drawing description information means converting the content that the application needs to display into a set of instructions that the rendering engine can understand.

[0050] The application parses the content that needs to be displayed into structured data, including attributes such as the position, style, and resource reference of the elements. Through logical code, the content is mapped into a data structure recognizable by the rendering engine, such as JSON, XML, or a custom format.

[0051] Abstracting complex display content into structured description information facilitates subsequent processing and rendering. It improves the maintainability and scalability of the code and facilitates dynamic adjustment of the display content.

[0052] Elements, including: drawing text, drawing images, drawing shapes, and status information.

[0053] Element attributes for drawing text, including:

[0054] The text content to be displayed; the position coordinates of the text in the image; the style attributes of the text, such as color, size, alignment, etc.; the font file resource: the font file used to render the text.

[0055] Element attributes for drawing images, including:

[0056] The position coordinates of the image in the image; the style attributes of the image, such as scaling, rotation, transparency, etc.

[0057] Element attributes for drawing shapes, including:

[0058] The type of shape to be drawn, such as rectangle, circle, line, etc.; the position coordinates of the shape in the image; the style attributes of the shape, such as fill color, border color, line thickness, etc.

[0059] Element attributes for status information, including:

[0060] Transformation operations applied to the drawn content, such as translation, rotation, scaling, etc.; defining the clipping area of the drawn content; relevant information about the view, such as viewport size, scaling ratio, etc.

[0061] Step 120, generate a picture according to the drawing description information.

[0062] The drawing description information usually exists in text form, such as shape, color, position, size, etc. The drawing description information can be parsed to convert the natural language description into instructions and parameters that the computer can understand.

[0063] It can be based on natural language processing technology and predefined rules or patterns. By means of lexical analysis, syntactic analysis, etc., the description information is disassembled into multiple parts. For example, for the drawing description information: draw a red circle with a radius of 50 in the upper left corner of the picture (coordinates (0,0)), the shape category of "circle" will be recognized, "red" as the color attribute, "(0,0)" as the center position of the circle, and "50" as the radius size.

[0064] Different graphic elements require different drawing algorithms. The appropriate drawing algorithm can be selected according to the parsed graphic type. For effects such as color filling and texture mapping, corresponding graphic processing technologies will also be called. For example, when filling color, the color value of each pixel will be determined according to the color model and written to the corresponding position in the image buffer.

[0065] Output the calculated graphic data as a picture according to the resolution and display format of the display device, and be able to accurately generate the picture desired by the user according to the given drawing description information.

[0066] The following will be described in conjunction with Figure 2 as follows. As Figure 2 shown, the application 102 process is used to convert the content to be displayed into drawing description information; and write the drawing description information and the drawing resources associated with the drawing description information into the shared resource allocator 101; the drawing resources may include: picture resources, font file resources, and synchronization control resources.

[0067] The application 102 process is further used to send a first notification message to the display service 103, and the first notification message is used to indicate that the drawing description information and the drawing resources have been written into the shared resource allocator 101.

[0068] The display service 103 process is used to obtain the drawing description information from the shared resource allocator 101 when receiving the first notification message; and calculate the redraw area according to the shared resource allocator.

[0069] The processor is used to perform rendering processing on the content to be displayed according to the redraw area, the drawing description information, and the drawing resources to generate a picture.

[0070] The display service is further used to output the picture to the display device.

[0071] The application process maps the content to be displayed into structured drawing description information, loads the drawing resources and writes them into the shared resource allocator, and can use the synchronization control resources to ensure data consistency. The application process informs the display service process through the first notification message that the drawing description information and the drawing resources are ready, and the first notification message can be sent using the inter-process communication mechanism.

[0072] After receiving the first notification message, the display service process reads the drawing description information and resources from the shared resource allocator and calculates the area that needs to be redrawn. Read data from the shared resource allocator. Analyze the drawing description information and calculate the redraw area.

[0073] The processor uses the drawing description information and the drawing resources to generate the final picture, uses the graphics API to render the description information into pixel data, and stores the rendering result in the frame buffer.

[0074] The display service process sends the rendering result to the display device and presents it to the user. Transmit the content of the frame buffer to the video memory of the display device. Trigger the refresh operation of the display device.

[0075] The application process efficiently converts the content to be displayed into instructions and data that can be understood by the rendering engine. It achieves efficient cooperation with the display service process through a shared resource allocator. The display service process optimizes the rendering performance by calculating the repainting area and reduces unnecessary drawing operations. It converts the abstract drawing description information into a specific visual output. The processor efficiently executes the rendering process, generates high-quality images, and supports complex graphics rendering requirements. The display device presents the rendering result to the user in real time to ensure a smooth visual experience.

[0076] In the existing graphics system, the application directly renders the content to generate an image frame buffer, that is, pixel data. The image frame buffer is the pixel data of a 2D image and occupies a large amount of memory. For example: for a 1920x1080 screen, each pixel occupies 4 bytes, and one frame of image requires about 8MB of memory. It occupies a relatively large memory space. Each application has its own image frame buffer, and as the number of applications increases, the memory occupancy will increase exponentially. Due to the independent rendering of applications, the time points of animations may be inconsistent, resulting in an uncoordinated overall display effect.

[0077] Exemplarily, in the existing graphics system, Application A and Application B respectively render their own content to generate image frame buffers. The animation of Application A is completed within 0.8 seconds, and the animation of Application B is completed within 1.2 seconds. The display service combines the two image frame buffers into the final picture, but due to the inconsistent animation times, the display effect is uncoordinated.

[0078] In the embodiments of the present application, using drawing description information instead of an image frame buffer can reduce memory occupancy and improve animation consistency. This is because the drawing description information is a kind of structured data that describes how to draw the content. For example: to draw a rectangle, only information such as position, size, and color needs to be stored, rather than storing the pixel data of the entire rectangle. The memory occupancy of the drawing description information is much smaller than that of the image frame buffer. Using the drawing description information can significantly reduce the occupancy of graphics memory, especially when running multiple applications.

[0079] In the embodiments of the present application, the drawing process only occurs in the display service process. The application only generates drawing description information and does not directly render the content. The display service performs unified rendering according to the drawing description information, and the time points of animations are more consistent. When the application and the display service execute animations simultaneously, more consistent display content can be obtained.

[0080] Exemplarily, through the embodiments of the present application, Application A and Application B respectively generate drawing description information and submit it to the display service. The display service performs unified rendering according to the drawing description information. The animations of Application A and Application B are synchronously completed within 1.0 second, and the display effect is more consistent.

[0081] Using drawing description information instead of the image frame buffer significantly reduces the occupancy of graphics memory. It is particularly suitable for scenarios where multiple applications are running, avoiding waste of memory resources. The display service performs unified rendering, and the time points of dynamic effects are more consistent. The dynamic effects of the application and the display service can be better synchronized, providing a smoother user experience. By reducing memory occupancy and invalid rendering, the load on the CPU and GPU is reduced. By introducing drawing description information, the deficiencies of the existing graphics system in terms of memory occupancy and dynamic effect consistency are addressed. Drawing description information saves more memory than the image frame buffer. Unified rendering by the display service ensures the consistency of dynamic effects.

[0082] In the embodiments of the present application, the application process no longer directly uses the graphics acceleration hardware to render the content to be displayed into the image frame buffer. Instead, it converts the content to be displayed into corresponding drawing description information and provides the drawing description information to the display service process. Finally, the display service process aggregates the drawing description information of each application process, completes the rendering and composition of the final display content according to the drawing description information, and presents the final screen on the display device.

[0083] In a possible embodiment, in step 120, it may specifically include the following steps:

[0084] Step 210, calculate the redraw area according to the drawing description information;

[0085] Step 220, generate a screen according to the drawing description information and the redraw area.

[0086] Drawing description information: Information used to describe the specific content of the screen to be generated. It may include detailed information such as the shape, color, position, size of elements, and the relationships between various elements in the screen.

[0087] Redraw area: Refers to a specific area where content needs to be redrawn during the process of generating a screen. For example, when only a local element in the screen is modified, the redraw area is only the local range where the element is located, rather than the entire screen.

[0088] First, parse the drawing description information and extract key information related to the position, size, etc. of the screen elements. For example, for the drawing description information of "drawing a circle with a radius of 30 at the coordinates (100, 100)", according to the position and radius information of the circle, calculate the rectangular area occupied by the circle in the screen, that is, the smallest rectangle containing the circle, and this rectangular area is a possible form of the redraw area.

[0089] If the drawing description information involves multiple elements, the positions and sizes of these elements can be comprehensively considered to calculate the area that can cover all relevant elements as the redrawing area. The purpose is to clarify the specific range that needs to be processed when generating the next frame, avoid unnecessary drawing operations on the entire frame, improve efficiency, and reduce the occupation of graphics processing resources.

[0090] After determining the redrawing area, the generation of the frame is carried out within this specific area according to the drawing description information. According to the definitions of the attributes of the elements such as shape, color, and texture in the description information, the frame content that meets the requirements is constructed pixel by pixel within the redrawing area.

[0091] Generating the frame within the redrawing area according to the drawing description information can ensure that the generated frame accurately meets the given description requirements. For the drawing description information containing dynamic elements, generating the frame in combination with the redrawing area can achieve dynamic updates of the frame. For example, in an animation scene, each time the frame is updated, by calculating the redrawing area, only the changed part is updated, and a new frame state is generated within this area according to the description information, so as to smoothly display the animation effect.

[0092] In a possible embodiment, there are multiple pieces of the drawing description information, and step 210 may specifically include the following steps:

[0093] Determine the window position information of each application window, and one piece of the drawing description information is associated with one application window;

[0094] According to the window position information, determine the window occlusion information between each application window;

[0095] According to the drawing description information and the window occlusion information, calculate the redrawing area.

[0096] Application window: An independent area where an application is displayed on the screen, usually containing its own content and drawing description information. Each window can independently manage its content and drawing logic.

[0097] Window position information: Describes the position and size of an application window on the screen, such as the upper left corner coordinates, width, and height. The window position information is used to determine the layout of the window and the spatial relationship between them.

[0098] Window occlusion information: Describes the occlusion relationship between multiple application windows, that is, which windows are partially or completely covered by other windows. The window occlusion information is used to optimize the drawing area and avoid drawing the occluded parts.

[0099] Redrawing area: The screen area that needs to be redrawn, usually one or more rectangular ranges. By limiting the drawing range, the rendering performance is optimized, and unnecessary full-screen redrawing is avoided.

[0100] Obtain the position and size information of each application window for subsequent occlusion analysis and redrawing area calculation. Obtain the coordinates and dimensions of each window from the window management system, where the coordinates are like (x, y) and the dimensions are like width and height. Associate the window position information with the drawing description information to ensure that the drawing description information of each window is consistent with its actual position. Ensure that the drawing description information of each window is consistent with its actual position, avoid misalignment of the drawn content, and provide basic data for subsequent occlusion analysis and redrawing area calculation.

[0101] By comparing the positions and sizes of windows, analyze which windows are occluded by other windows. Use geometric algorithms to judge the overlapping relationship between windows, such as rectangle intersection detection, and record the visible area and occluded area of each window. By analyzing the occlusion relationship between windows, avoid drawing completely occluded areas and reduce unnecessary rendering calculations.

[0102] Combine the drawing description information and window occlusion information to determine which areas need to be redrawn. Traverse the drawing description information of each window to determine the area that needs to be drawn. According to the window occlusion information, exclude the areas that are completely occluded by other windows. Merge the visible areas of all windows to obtain the final redrawing area.

[0103] By excluding occluded areas, optimize the drawing range, reduce the load on the GPU and CPU. When there is a dynamic window layout, it can efficiently update the screen and improve the user experience. Support complex multi-window scenarios to ensure that the content of each window is correctly displayed.

[0104] By combining the window position information and occlusion information, the calculation of the redrawing area is optimized, which is applicable to multi-window scenarios, can significantly improve the rendering efficiency, reduce resource waste, and at the same time ensure that the content of each window is correctly displayed.

[0105] In one possible embodiment, in the step of determining the window position information of each application window, it may specifically include the following steps:

[0106] Obtain each of the application windows through the window manager and set the window position information of each of the application windows;

[0107] The determining the window occlusion information between each of the application windows according to the window position information includes:

[0108] Send the window position information to the display service process through the window manager for the display service process to calculate the window occlusion information between each of the application windows according to the window position information.

[0109] Window Manager: A component in an operating system or graphical system responsible for managing application windows, which is responsible for window creation, destruction, layout, and display order, etc. It coordinates the display of multiple application windows to ensure correct stacking and interaction between windows.

[0110] Display Service Process: A system process or service responsible for finally rendering the window content to the screen. It receives instructions from the window manager, calculates the occlusion relationship between windows, and generates the final display image.

[0111] The window manager is responsible for managing the layout and display order of all application windows and obtaining the position information of each window through the window manager. The window manager maintains a window list that records the coordinates and dimensions of each window. When the window position information changes, the window manager updates the window position information.

[0112] In a scenario where multiple windows are stacked and displayed, the window manager needs to pass the window position information to the display service process for calculating the occlusion relationship between windows. The window manager sends the window position information to the display service process through an inter-process communication mechanism, such as a message queue, shared memory, or remote procedure call.

[0113] The display service process receives the window position information and prepares for occlusion analysis. The display service process analyzes the occlusion relationship between windows based on the received window position information. It uses geometric algorithms to determine the overlapping relationship between windows, records the visible and occluded areas of each window, and generates occlusion information.

[0114] Obtaining and setting the window position information can ensure that the window manager can accurately manage the layout and display order of each window, providing basic data for subsequent occlusion analysis and redrawing area calculation. Sending the window position information to the display service process can enable the collaborative work between the window manager and the display service process, ensuring that the occlusion information can be updated in real time. By analyzing the occlusion relationship between windows, it is possible to avoid drawing completely occluded areas, reducing unnecessary rendering calculations, improving the rendering efficiency, especially in a scenario where multiple windows are stacked and displayed, ensuring the correct display of the content of each window, and enhancing the user experience.

[0115] Through the collaborative work of the window manager and the display service process, efficient occlusion analysis and redrawing area calculation in a multi-window scenario are achieved, which can significantly improve the rendering efficiency, reduce resource waste, and at the same time ensure the correct display of the content of each window.

[0116] Such as Figure 3As shown, in a graphics system based on server-side rendering, the display service can optimize the calculation of the redraw area according to the window position information of the window manager 107. The display service receives the drawing description information of each application. The drawing description information describes the content that the application wants to display.

[0117] The display service calculates the occlusion relationship between the windows of each application according to the window positions and hierarchical relationships provided by the window manager. For example: Application window B is located above application window A and partially occludes the content of application window A. The display service combines the changes in the drawing description information and the window occlusion relationship to calculate a more accurate redraw area. For example: If the cloud in application window A moves to the area occluded by application window B, the display service can determine that this part of the content does not need to be drawn, thereby reducing the redraw area.

[0118] As Figure 4 shown, in the existing graphics system, the application renders independently to generate an image frame buffer. The application does not know whether its content is occluded by other windows during rendering. For example: When the cloud in application window A moves to the area occluded by application window B, application window A still renders the entire area where the cloud changes, even though this part of the content will not be seen by the user. This results in ineffective drawing and wastes system resources.

[0119] In the embodiments of the present application, the display service uniformly manages the rendering, and the display service can determine the occlusion relationship between each window. The display service can determine that the occluded area does not need to be drawn, thereby calculating a smaller redraw area. This reduces ineffective drawing and improves system performance.

[0120] As Figure 5 shown, Application window A: Displays a sky and a cloud. Application window B: Displays a dialog box that partially occludes the cloud in application window A. The cloud in application window A moves to the right and enters the area occluded by application window B.

[0121] The display service determines, according to the change in the drawing description information, that the cloud moving to the occluded area does not need to be drawn. The display service only renders the unoccluded part of the cloud, calculating a smaller redraw area. This avoids ineffective drawing and improves system performance.

[0122] The display service calculates a more accurate redraw area based on the window occlusion relationship and the drawing description information. This avoids ineffective drawing of the occluded area, saving CPU and GPU resources. Improving rendering efficiency: By reducing the redraw area, the rendering load is reduced, and system performance is improved. It is especially suitable for complex scenarios with multiple windows overlapping for display. Optimizing the user experience: A more efficient rendering mechanism can provide smoother animation effects and faster response speeds. It reduces the waste of system resources and extends the battery life of mobile devices.

[0123] In a possible embodiment, drawing resources are determined according to the drawing description information.

[0124] Step 220 may specifically include the following steps:

[0125] A picture is generated according to the drawing description information, the drawing resources, and the redrawing area.

[0126] Drawing resources: Resources required during the drawing process, providing specific data support for the rendering engine during drawing. Among them, drawing resources include: picture resources, font file resources, and synchronization control resources.

[0127] Extract the required resource references from the drawing description information, such as picture paths, font names, etc., and load these resources for rendering use. Load picture resources through the resource management module, such as loading pictures from disk or network; load font file resources, such as loading fonts from the system font library or custom font files. Initialize synchronization control resources, such as creating locks or semaphores, to ensure data consistency during multi-threaded drawing.

[0128] Ensure that the rendering engine can access all necessary resources to avoid resource shortages or errors during the drawing process. Through synchronization control resources, support multi-threaded or asynchronous drawing, improve rendering efficiency, and avoid resource competition problems.

[0129] Specifically, the application can determine the drawing resources associated with the drawing description information. The main task of the application is to convert the content to be displayed into drawing description information, write the relevant resources into the shared resources, and finally notify the display service that it can start rendering. The specific steps are as follows:

[0130] The application needs to convert the content to be displayed into drawing description information. The drawing description information, for example:

[0131] The drawing description information corresponding to drawing a rectangle includes: position, size, color, transparency, etc. The drawing description information corresponding to drawing a picture includes: picture resource ID, position, scaling ratio, etc. The drawing description information corresponding to drawing a piece of text includes: font resource ID, text content, position, color, etc.

[0132] The application writes the generated drawing description information into the shared resources. The shared resources are managed by a shared resource allocator, and both the application and the display service can access these resources. The drawing description information may reference other resources, for example:

[0133] Picture resources: Pictures that the application needs to display.

[0134] Font file resources: Fonts required by the application.

[0135] The application also needs to write these resources into the shared resources to ensure that the display service can access them. When the drawing description information and related resources are ready, the application notifies the display service that the shared resources are ready. The display service can start accessing the shared resources for rendering.

[0136] Based on the drawing description information, drawing resources, and the redraw area, an image to be displayed on the screen is finally generated. The abstract description information is converted into a specific visual output. The rendering engine generates the screen according to the instructions in the drawing description information, in combination with the loaded resources and the defined redraw area.

[0137] Specifically, the display service can calculate the redraw area according to the drawing description information. The display service receives a notification from the application, knowing that the drawing description information in the shared resources is ready. The display service accesses the shared resources to read the drawing description information.

[0138] The display service can calculate the area that needs to be redrawn in the way of the existing graphics system. For example, if only the content of the window of one application has changed, the display service only needs to redraw the area where the window is located, rather than the entire screen.

[0139] The display service completes the actual rendering work according to the drawing description information, using the CPU or graphics acceleration hardware. If the drawing description information contains complex graphics operations, such as 3D rendering, texture mapping, etc., the display service will give priority to using the GPU to accelerate the rendering. If the drawing description information only contains simple graphics operations, such as 2D geometric graphics drawing, the display service may use the CPU to complete the rendering.

[0140] If the content of multiple applications needs to be displayed, the display service will synthesize them into the final picture. When synthesizing, the display service will determine how to overlay the content according to the occlusion relationship of the layers. The final picture will be output to the display device.

[0141] Exemplarily, Application A and Application B are running simultaneously:

[0142] The window of Application A is located on the left side of the screen, showing a picture.

[0143] The window of Application B is located on the right side of the screen, showing a piece of text.

[0144] Application A updates the picture content, while the content of Application B remains unchanged.

[0145] The display service receives the drawing description information of application A, calculates the repainting area, and finds that only the window area of application A needs to be redrawn. The new picture content of application A is rendered using the GPU. The contents of application A and application B are synthesized into the final picture. The final picture is output to the display device. By limiting the repainting area, the number of pixels that the rendering engine needs to process is reduced, thereby improving the rendering performance. Avoid full-screen repainting, reduce the load on the GPU and CPU, and improve the response speed and smoothness of the application. When dynamic content is updated, the screen can be efficiently updated locally, enhancing the user experience.

[0146] In a possible embodiment, through the application process, the drawing resources associated with the drawing description information are determined and written to the shared resource allocator, and the application runs on the server;

[0147] Through the display service process, the drawing description information is obtained from the shared resource allocator, and the repainting area is calculated according to the drawing description information; the display service runs on the electronic device.

[0148] Shared resource allocator: A mechanism or component for sharing resources among multiple processes, such as shared memory, message queues, etc. It allows efficient transfer of drawing description information and drawing resources between the application process and the display service process.

[0149] The application process generates the drawing description information and associated drawing resources, and passes this data to the display service process through the shared resource allocator. The application process loads the drawing resources locally or remotely. These resources are written to the shared resource allocator together with the drawing description information to ensure the efficiency and consistency of data transfer between processes.

[0150] The above steps can achieve efficient data transfer between the application process and the display service process, reducing communication overhead. It supports a distributed rendering architecture between the server and the electronic device, and is suitable for remote display or cloud computing scenarios.

[0151] The display service process reads the drawing description information and associated drawing resources from the shared resource allocator. Analyze the element positions and ranges in the drawing description information, and calculate the repainting area for each element. Merge the repainting areas of all elements to obtain the final repainting area.

[0152] Obtaining the drawing description information and resources through the shared resource allocator ensures that the rendering engine can access all necessary data. By calculating the repainting area, the rendering performance is optimized, and unnecessary full-screen repainting is avoided. When dynamic content is updated, the screen can be efficiently updated locally, enhancing the user experience.

[0153] Efficient collaboration between application processes and display service processes is achieved through a shared resource allocator, which is particularly suitable for distributed rendering scenarios, can significantly improve rendering efficiency, reduce communication overhead, and at the same time support complex graphics display requirements.

[0154] As Figure 6 shown, in a distributed scenario, the application and the display service can be distributed on different devices. For example:

[0155] Application 102 runs on device A, such as a cloud server. The display service 103 runs on device B, such as a terminal device, and is responsible for displaying the content to the user. In this scenario, the shared resource allocator 101 can be deployed on different devices, that is, deployed on device A and device B respectively. The application and the display service can still apply for and access resources through the shared resource allocator. The shared resource allocators can complete cross-network transmission and synchronization of shared resources.

[0156] The drawing description information and resources generated by the cloud application can be transmitted over the network to the shared resource allocator of the edge device for use by the display service. The application and the display service still work according to the working principle of a single device. The application generates drawing description information and resources, and the display service renders according to this information.

[0157] Existing cross-device display graphics systems, such as streaming display, usually adopt the following method: The cloud application encodes the rendering result into a video stream and transmits it over the network to the edge device. The edge device decodes the video stream and displays the content. The image frame buffer is pixel data and occupies a large amount of bandwidth. For example: A frame of image with a resolution of 1920x1080 requires about 8MB of data. In the case of high resolution and high frame rate scenarios, the network bandwidth pressure is huge. The encoding, transmission, and decoding processes of the video stream will introduce additional latency. For applications with high real-time requirements, the latency will affect the user experience. Even if the content has not changed, the streaming display still needs to transmit the complete image frame buffer, resulting in waste of resources.

[0158] The drawing description information is structured data that describes how to draw the content. Compared with the image frame buffer, the amount of data of the drawing description information is much smaller. For example: To draw a rectangle, only information such as position, size, and color needs to be transmitted, rather than transmitting the pixel data of the entire rectangle. Due to the reduction in data transmission volume, the pressure on network bandwidth is reduced, and the display frame rate can be increased. It is particularly suitable for high resolution and high frame rate scenarios. The transmission and processing speed of the drawing description information is faster, reducing the latency of encoding, decoding, and transmission. For applications with high real-time requirements, the latency is significantly reduced.

[0159] For example, in a game scenario, a 3D game runs on a cloud server, and the user displays the game screen through a mobile phone. The cloud server generates rendering description information, such as the transformation, texture, and lighting of the 3D model. The rendering description information is transmitted to the mobile phone through the network. The display service of the mobile phone renders the game screen according to the rendering description information. The data transmission volume is small, reducing bandwidth pressure. The latency is low, providing a smoother gaming experience. If the game screen does not change, only the changed rendering description information needs to be transmitted.

[0160] In distributed scenarios, drawing description information saves more bandwidth than image frame buffers, reduces network bandwidth pressure, supports higher frame rate display, reduces encoding, transmission and decoding delays, improves real-time performance, and only transmits changed drawing description information to avoid invalid data transmission. Compared with existing streaming displays, systems based on drawing description information are more suitable for distributed scenarios and can provide a more efficient and smoother cross-end display experience.

[0161] In the embodiment of the present application, by converting the content to be displayed into drawing description information; the drawing description information is structured data used to describe the element attributes of the elements in the content to be displayed; the drawing description information is used to replace the image frame buffer, and the memory occupation of the drawing description information is much smaller than that of the image frame buffer, which can reduce the memory occupation. According to the drawing description information, the picture is generated, which can reduce the graphics memory overhead.

[0162] The screen generation method provided in the embodiment of the present application can be executed by a screen generation device. In the embodiment of the present application, the screen generation device provided in the embodiment of the present application is described by taking the screen generation method executed by the screen generation device as an example.

[0163] Figure 7 700 is a block diagram of a screen generating device provided in an embodiment of the present application, the device 700 includes:

[0164] The conversion module 710 is used to convert the content to be displayed into drawing description information; the drawing description information is structured data used to describe the element attributes of the elements in the content to be displayed;

[0165] The generating module 720 is used to generate a picture according to the drawing description information.

[0166] In a possible embodiment, the generating module 720 includes:

[0167] A calculation module, used for calculating a redrawing area according to the drawing description information;

[0168] The generating module 720 is specifically used to generate a picture according to the drawing description information and the redrawing area.

[0169] In a possible embodiment, the computing module includes:

[0170] A first determination module, configured to determine the window position information of each application window, and one piece of the drawing description information is associated with one of the application windows;

[0171] A second determination module, configured to determine the window occlusion information between each of the application windows according to the window position information;

[0172] The calculation module is specifically configured to:

[0173] Calculate a redrawing area according to the drawing description information and the window occlusion information.

[0174] In a possible embodiment, the first determination module is specifically configured to:

[0175] Obtain each of the application windows through a window manager, and set the window position information of each of the application windows;

[0176] The second determination module is specifically configured to:

[0177] Send the window position information to a display service process through the window manager, so that the display service process calculates the window occlusion information between each of the application windows according to the window position information.

[0178] In a possible embodiment, the apparatus 700 further includes:

[0179] A third determination module, configured to determine drawing resources according to the drawing description information;

[0180] The generation module 720 is specifically configured to:

[0181] Generate a picture according to the drawing description information, the drawing resources, and the redrawing area.

[0182] In the embodiments of the present application, by converting the content to be displayed into drawing description information; the drawing description information is structured data for describing the element attributes of the elements in the content to be displayed; using the drawing description information instead of the image frame buffer, the memory occupation of the drawing description information is much smaller than that of the image frame buffer, which can reduce the memory occupation. Generating a picture according to the drawing description information can reduce the graphics memory overhead.

[0183] The screen generation device in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than terminals. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0184] The screen generation device in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0185] The screen generation device provided in the embodiments of the present application can implement each process implemented in the above method embodiments. To avoid repetition, it will not be elaborated here.

[0186] Optionally, as Figure 8 shown, the embodiments of the present application further provide an electronic device 810, including a processor 811, a memory 812, and a program or instruction stored on the memory 812 and executable on the processor 811. When the program or instruction is executed by the processor 811, it implements each step of any of the above screen generation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0187] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0188] Figure 9 Schematic diagram of the hardware structure of an electronic device for implementing the embodiments of the present application.

[0189] The electronic device 600 includes, but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and other components.

[0190] Those skilled in the art can understand that the electronic device 600 may further include a power supply (such as a battery) for powering each component. The power supply can be logically connected to the processor 610 through a power management system, so as to manage functions such as charging, discharging, and power consumption management through the power management system. Figure 9 The structure of the electronic device shown does not limit the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0191] Among them, the processor 610 is used to convert the content to be displayed into drawing description information; the drawing description information is structured data for describing the element attributes of the elements in the content to be displayed;

[0192] The processor 610 is further used to generate a picture according to the drawing description information.

[0193] Optionally, the processor 610 is further used to calculate a redrawing area according to the drawing description information;

[0194] The processor 610 is further used to generate a picture according to the drawing description information and the redrawing area.

[0195] Optionally, the processor 610 is further used to determine the window position information of each application window, and one drawing description information is associated with one application window;

[0196] The processor 610 is further used to determine the window occlusion information between each application window according to the window position information;

[0197] The processor 610 is further used to calculate a redrawing area according to the drawing description information and the window occlusion information.

[0198] Optionally, the processor 610 is further used to obtain each application window through a window manager and set the window position information of each application window;

[0199] The processor 610 is further used to send the window position information to a display service process through the window manager, so that the display service process calculates the window occlusion information between each application window according to the window position information.

[0200] Optionally, the processor 610 is further configured to determine rendering resources according to the rendering description information;

[0201] The processor 610 is further configured to generate a picture according to the rendering description information, the rendering resources, and the redrawing area.

[0202] In the embodiments of the present application, by converting the content to be displayed into rendering description information; the rendering description information is structured data for describing the element attributes of the elements in the content to be displayed; using the rendering description information instead of the image frame buffer, the memory occupancy of the rendering description information is much smaller than that of the image frame buffer, which can reduce the memory occupancy. Generating a picture according to the rendering description information can reduce the graphics memory overhead.

[0203] It should be understood that, in the embodiments of the present application, the input unit 604 may include a Graphics Processing Unit (GPU) 6041 and a microphone 6042. The graphics processor 6041 processes the image data of static pictures or video images obtained by an image capturing device (such as a camera) in the video image capturing mode or the image capturing mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also referred to as a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. The other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an action bar, which will not be elaborated here. The memory 609 may be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 610 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interfaces, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above modem processor may not be integrated into the processor 610.

[0204] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 609 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0205] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 610 either.

[0206] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned embodiment of the screen generation method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0207] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0208] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the screen generation method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0209] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0210] The embodiments of the present application provide a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above embodiment of the screen generation method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0211] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0212] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0213] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A screen generation method, characterized in that: The method comprises: Converting the content to be displayed into drawing description information; the drawing description information is structured data for describing element attributes of elements in the content to be displayed; A picture is generated according to the drawing description information.

2. The method according to claim 1, characterized in that The step of generating a picture according to the drawing description information includes: Calculate the redrawing area according to the drawing description information; A picture is generated according to the drawing description information and the redrawing area.

3. The method according to claim 2, characterized in that The drawing description information is multiple, and the calculating the redrawing area according to the drawing description information includes: Determine the window position information of each application window, and associate one of the drawing description information with one of the application windows; Determining window occlusion information between the application windows according to the window position information; A redrawing area is calculated according to the drawing description information and the window occlusion information.

4. The method according to claim 3, characterized in that The determining of the window position information of each application window includes: Acquire each of the application windows through a window manager, and set window position information of each of the application windows; The determining, according to the window position information, window occlusion information between the application windows includes: The window position information is sent to the display service process through the window manager, so that the display service process calculates the window occlusion information between the application windows according to the window position information.

5. The method according to claim 2, characterized in that: The method further comprises: Determining drawing resources according to the drawing description information; The step of generating a picture according to the drawing description information and the redrawing area includes: A picture is generated according to the drawing description information, the drawing resources and the redrawing area.

6. A screen generating device, characterized in that: The device comprises: A conversion module, used to convert the content to be displayed into drawing description information; the drawing description information is structured data used to describe element attributes of elements in the content to be displayed; A generating module is used to generate a picture according to the drawing description information.

7. The device according to claim 6, characterized in that The generating module comprises: A calculation module, used for calculating a redrawing area according to the drawing description information; The generating module is specifically used to generate a picture according to the drawing description information and the redrawing area.

8. The device according to claim 7, characterized in that The computing module comprises: A first determining module, used to determine the window position information of each application window, wherein one piece of drawing description information is associated with one application window; A second determining module, configured to determine window occlusion information between the application windows according to the window position information; The computing module is specifically used for: A redrawing area is calculated according to the drawing description information and the window occlusion information.

9. The device according to claim 8, characterized in that The first determining module is specifically configured to: Acquire each of the application windows through a window manager, and set window position information of each of the application windows; The second determining module is specifically used to: The window position information is sent to the display service process through the window manager, so that the display service process calculates the window occlusion information between the application windows according to the window position information.

10. The device according to claim 7, characterized in that The device also includes: A third determining module, configured to determine a drawing resource according to the drawing description information; The generation module is specifically used for: A picture is generated according to the drawing description information, the drawing resources and the redrawing area.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the screen generation method according to any one of claims 1 to 5 are implemented.

Images