A rendering and screen display method, device, equipment and medium

Abstract

The present application discloses a rendering and screen display method, apparatus, device and medium, which relates to the field of cloud games. The method is applied to a client system and includes: determining current rendering data and sending the rendering data to a direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data; determining target screen display data from the rendering data through the direct rendering manager by using a preset screen display optimization strategy; merging the target screen display data with the current layer, and sending the merged layer to a preset display screen for display. In this way, this solution uses the direct rendering manager to uniformly manage the three types of rendering data, avoiding the screen display conflict under the underlying framework in the prior art with a streamlined rendering process. Moreover, this solution also proposes a screen display optimization strategy, which can ensure the high frame rate performance of the client.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud games, and particularly relates to a rendering and screen display method, device, equipment and medium. Background Art

[0002] With the development of 5G networks, codec technologies, and software and hardware technologies, innovative services such as cloud game systems have gradually come into the public eye and achieved certain development. Different from traditional games, cloud games are a game implementation method based on cloud computing, cloud control, rendering, and network transmission. The computing power is uplinked, and the game actually runs on the host side of the cloud edge computing node, rather than on the user's local client. The client presents the game sound and picture locally by receiving the audio and video data sent by the host side. At the same time, the user sends the operation control data of the game to the host side to control the game.

[0003] To promote the development of cloud game systems, cloud game streaming terminals support multiple platforms, and the client terminals also include support for the linux platform. The client needs to render and synthesize data such as videos, UI (i.e., User Interface), and cursors for display on the screen. In the prior art, there are various open-source libraries in the linux system for rendering and displaying UI, videos, and cursors respectively. For example, common open-source C / C++ GUI (i.e., Graphical User Interface) libraries under linux include IMGUI, CEGUI, GTK, QT, etc. There are different players for displaying videos, and there are also corresponding technical solutions for cursors. However, the underlying frameworks used by these different open-source libraries may be different. Some use devfb, and some may use DRM (i.e., Direct Rendering Manager), etc. If the corresponding open-source libraries are used separately when displaying UI, videos, and cursors, various conflicts will occur during screen display. In addition, the rendering of common GUI libraries ultimately requires window processing, and generally uses the relatively large SDL (Simple DirectMedia Layer) graphics library for support. This includes too much content, and the graphics library is too bloated, resulting in a relatively high latency of cloud games and a poor experience effect.

[0004] As can be seen from the above, in the process of client rendering, how to avoid the situation of screen display conflicts caused by the existing rendering architecture and the poor experience effect of cloud games is a problem to be solved in this field. Summary of the Invention

[0005] In view of this, the object of the present invention is to provide a rendering and screen display method, device, equipment and medium, which can optimize the rendering underlying architecture, uniformly manage all video data, UI data and cursor data, and avoid screen display conflicts. In addition, this solution also simplifies some modules of the architecture, greatly improving the performance of the linux cloud game client and promoting the development of the cloud game system. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a rendering and screen display method, which is applied to a client system and includes:

[0007] Determine the current rendering data and send the rendering data to a direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data;

[0008] Through the direct rendering manager and using a preset screen display optimization strategy, determine target screen display data from the rendering data;

[0009] Merge the target screen display data with the current layer and send the merged layer to a preset display screen for display.

[0010] Optionally, the determining the current rendering data includes:

[0011] Generate frame buffer data by using a first interface pre-connected to a graphical user interface and a second interface pre-connected to an open graphics library;

[0012] Obtain the frame buffer data by using a preset third interface, and use a preset general buffer manager to convert the frame buffer data into frame buffer data with a preset identifier; the preset identifier is an identifier recognizable by the direct rendering manager;

[0013] Use the frame buffer data with the preset identifier as the current user interface rendering data.

[0014] Optionally, the determining the current rendering data includes:

[0015] Obtain cursor pixel data sent by the host side; the cursor pixel data is data representing the pixel values of the cursor image;

[0016] Based on the rendering pixel data, determine cursor rendering data by using a preset cursor thread; the cursor rendering data includes the position coordinate data of the cursor, the resolution data of the cursor, and the color mode data of the cursor.

[0017] Optionally, the determining the current rendering data includes:

[0018] Obtain a video bitstream sent by the host side;

[0019] Perform hardware decoding on the video bitstream to obtain video data corresponding to the video bitstream, and use the video data as video rendering data.

[0020] Optionally, the step of determining target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy further includes:

[0021] When the direct rendering manager receives the video rendering data, store the video rendering data in a preset stack;

[0022] Determine target on-screen data from all the video rendering data in the preset stack.

[0023] Optionally, the rendering on-screen method further includes:

[0024] Determine the number of data of the video rendering data in the stack;

[0025] If the number of data of the video rendering data in the stack is greater than 1, determine the latest video rendering data from all the video rendering data, and pop the video rendering data other than the latest video rendering data from all the video rendering data.

[0026] Optionally, the rendering on-screen method further includes:

[0027] Obtain the on-screen cycle through a preset time acquisition interface;

[0028] Correspondingly, the step of merging the target on-screen data with the current layer and sending the merged layer to a preset display screen for display includes:

[0029] Based on the on-screen cycle, merge the target on-screen data with the current layer, and send the merged layer to a preset display screen for display.

[0030] In a second aspect, the present application discloses a rendering on-screen device, which is applied to a client system and includes:

[0031] A rendering data determination module, configured to determine current rendering data and send the rendering data to a direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data;

[0032] An on-screen data determination module, configured to determine target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy;

[0033] The upper screen display module is used to merge the target upper screen data with the current layer and send the merged layer to a preset display screen for display.

[0034] In a third aspect, the present application discloses an electronic device, including:

[0035] A memory for storing a computer program;

[0036] A processor for executing the computer program to implement the foregoing rendering and upper screen method.

[0037] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the foregoing disclosed rendering and upper screen method are implemented.

[0038] In the present application, the current rendering data is first determined and sent to the direct rendering manager in the client system. The rendering data includes user interface rendering data, cursor rendering data, and video rendering data. Then, through the direct rendering manager and using a preset upper screen optimization strategy, target upper screen data is determined from the rendering data. Next, the target upper screen data is merged with the current layer, and the merged layer is sent to a preset display screen for display. In this way, after the current rendering data is determined in this solution, the rendering data is sent to the direct rendering manager in the client system so that the direct rendering manager can uniformly manage these rendering data, and the streamlined rendering process avoids the upper screen conflict in the prior art under the underlying framework. In addition, based on the preset upper screen optimization strategy, the target upper screen data is determined in this solution, which can ensure that the latest video rendering data is used for display each time the upper screen is displayed, so as to complete the optimization of the upper screen. Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0040] Figure 1 It is a flowchart of a rendering and upper screen method provided by the present application;

[0041] Figure 2 It is a schematic diagram of the process of an upper screen optimization strategy provided by the present application;

[0042] Figure 3 It is a flowchart of a method for determining a specific UI data provided by the present application;

[0043] Figure 4 A schematic diagram of the workflow of a thread provided for this application;

[0044] Figure 5 A schematic diagram of data rendering proposed for this application;

[0045] Figure 6 A schematic diagram of the structure of a rendering-on-screen device proposed for this application;

[0046] Figure 7 A structural diagram of an electronic device provided for this application. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0048] In the prior art, the client may use different open-source libraries and different underlying frameworks for UI data, video data, and cursor data, resulting in various conflicts when rendering on the screen. In this application, the rendering underlying architecture is optimized to uniformly manage all video data, UI data, and cursor data to avoid conflicts when rendering on the screen. In addition, this solution also simplifies some modules of the architecture, greatly improving the performance of the linux cloud game client and promoting the development of the cloud game system.

[0049] An embodiment of the present invention discloses a method for rendering on the screen, which is applied to a client system. Refer to Figure 1 As described, the method includes:

[0050] Step S11: Determine the current rendering data and send the rendering data to the direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data.

[0051] In the specific application process of cloud games, the client receives the audio and video data sent by the host to present the game sound and picture locally, and the user sends the game operation control data to the host to control the game. In other words, the host is the end that provides services to the client, and the client is the end that is served. When the client receives the relevant data sent by the host, it will use these data to render the client so that these data can be synthesized on the screen and then presented on the local display. These data may include user interface rendering data (hereinafter referred to as UI data), cursor rendering data, and video rendering data. Since the most important data in the cloud game streaming is video data, the performance of the video will be given priority. That is to say, in some specific implementations, among these rendering data, video rendering data has a higher priority than other data.

[0052] In this embodiment, the determining of the current rendering data includes determining the current UI data. In this embodiment, the generated frame buffer data may be determined as the current UI data by using some preset modules through the UI thread.

[0053] In this embodiment, determining the current rendering data may include: obtaining cursor pixel data sent by the host end; the cursor pixel data is data representing the pixel value of the cursor image; based on the rendering pixel data, and using a preset cursor thread to determine the cursor rendering data; the cursor rendering data includes the cursor position coordinate data, the cursor resolution data and the cursor color mode data.

[0054] It can be understood that in the application process of cloud games, the host side is the party that provides cloud game services to the client side. The host side will first transmit the entire cursor pixel data to the client side. After the client receives the cursor pixel data, the cursor position coordinates (x, y), cursor resolution size (w, h) and cursor argb data are sent to the direct rendering manager using the cursor thread. In a specific implementation, these cursor rendering data can be directly updated to the direct rendering manager.

[0055] In this embodiment, determining the current rendering data may include: obtaining a video stream sent by the host end; performing hardware decoding on the video stream to obtain video data corresponding to the video stream, and using the video data as video rendering data.

[0056] It can be understood that in the application process of cloud games, the host side will first encode the video into a video stream and transmit the video stream to the client. After the client receives the video stream, it will perform hardware decoding on the video. After decoding it into the corresponding video, it will determine the corresponding video rendering data based on the video and send the video rendering data to the direct rendering manager. In a specific implementation, the video rendering data can be directly updated to the direct rendering manager.

[0057] In a specific implementation, after the UI data, cursor rendering data, and video rendering data are updated in the direct rendering manager, when the direct rendering manager detects corresponding changes to the UI data, cursor data, and video data, it determines the target on-screen data based on the current rendering data, and submits the target on-screen data to synchronize the screen using the target on-screen data.

[0058] Step S12: determining target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy.

[0059] In this embodiment, the target on-screen data is determined from the rendering data through the direct rendering manager and using the preset on-screen optimization strategy, and it can also include: when the direct rendering manager receives the video rendering data, the video rendering data is stored in a preset stack; correspondingly, the target on-screen data is determined from the rendering data, including: determining the target on-screen data from all the video rendering data in the preset stack. It can be understood that in this embodiment, a stack is created for the video rendering data to store the received video rendering data, and the target on-screen data is determined from these video rendering data. It should be pointed out that after the target on-screen data is determined, the above-mentioned target on-screen data will be submitted to synthesize the target on-screen data with the current layer, and display it on the display.

[0060] In this embodiment, the method for rendering on the screen may further include: obtaining the on-screen period through a preset time acquisition interface; accordingly, the use of the target on-screen data to merge with the current layer and sending the merged layer to the preset display screen for display may include: based on the on-screen period and using the target on-screen data to merge with the current layer, and sending the merged layer to the preset display screen for display. It can be understood that in this embodiment, the on-screen period is also set for the client, that is, in the process of using the rendering data for on-screen display, the screen on the display screen will be updated according to the on-screen period.

[0061] In this embodiment, the rendering and screen display method may further include: determining the number of pieces of video rendering data in the stack; if the number of pieces of video rendering data in the stack is greater than 1, determining the latest video rendering data from all the video rendering data, and popping the video rendering data other than the latest video rendering data from all the video rendering data. It can be understood that when the number of pieces of video rendering data in the stack is greater than 1, it means that multiple pieces of video rendering data are received. Since the stack-in rule is push at the tail, it means that these video rendering data are stored in the stack in chronological order. At this time, the latest video rendering data can be determined from the stack for screen display, and the remaining video rendering data are popped from the stack and discarded to complete the optimization of the screen display data. It can be understood that if the number of pieces of video rendering data in the stack is 1, it can be directly determined as the target screen display data.

[0062] It should be noted that after the target screen display data is submitted, the submitted target screen display data will be popped from the stack. That is to say, when the number of pieces of video rendering data in the stack is 1 and it is determined as the target screen display data, it will be popped from the stack; when the number of pieces of video rendering data in the stack is greater than 1 and the latest video rendering data is determined as the target screen display data and submitted, it will also be popped from the stack. As Figure 2 shown in a schematic diagram of the process of a screen display optimization strategy. The figure shows that after the video is hard-decoded, the video rendering data FB0, FB1, FB2, FB3, FBX are stored in a preset stack in a push-at-the-tail manner, and then the target screen display data is taken out of the stack for rendering and screen display. When there is only one piece of video rendering data in the stack, it is popped from the stack and displayed on the screen; when there are multiple pieces of video rendering data in the stack, the target screen display data for screen display is determined according to the cached frame situation and display period of the data in the stack, and the remaining video rendering data are popped from the stack and discarded; when there are multiple pieces of video rendering data in the stack and the decoder is in the decoding state, it means that there is a latest video being decoded in addition to the data in the stack. Then, all the video rendering data in the stack are popped from the stack and discarded, and the latest frame is determined as the target screen display data for screen display. That is, the target screen display data for screen display in this embodiment is always the current latest video frame data.

[0063] It can be understood that an upper-screen optimization strategy is proposed in this embodiment. After receiving the video rendering data, the video rendering data can be stored in the stack in a push-at-the-tail manner, and the latest rendering data in the stack is determined as the target upper-screen data to ensure that the latest video rendering data is used for display every time the screen is displayed, which can guarantee the high frame rate performance of the client to complete the optimization of the upper screen.

[0064] Step S13: Merge the target on-screen data with the current layer, and send the merged layer to a preset display screen for display.

[0065] In this embodiment, by first determining the current rendering data and sending the rendering data to the direct rendering manager in the client system, the rendering data includes user interface rendering data, cursor rendering data, and video rendering data, and then through the direct rendering manager and using a preset on-screen optimization strategy, the target on-screen data is determined from the rendering data. Then, the target on-screen data is merged with the current layer, and the merged layer is sent to a preset display screen for display. In this way, after determining the current rendering data, this solution sends the rendering data to the direct rendering manager in the client system so that the direct rendering manager can uniformly manage these rendering data, avoiding the on-screen conflicts in the underlying framework in the prior art. In addition, an on-screen optimization strategy is proposed in this solution, which can ensure that the latest video rendering data is used for display each time on-screen display is performed to complete the optimization of on-screen display and maintain the high frame rate performance of the client.

[0066] Figure 3 It is a flowchart of a specific method for determining UI data provided by an embodiment of the present application. Refer to Figure 3 As shown, the method includes:

[0067] Step S21: Generate frame buffer data by using a first interface pre-connected to a graphical user interface and a second interface pre-connected to an open graphics library.

[0068] In a specific implementation manner, the first interface pre-connected to the graphical user interface may adopt the API (i.e., Application Programming Interface) of IMGUI, or may also be the API interface of other UI libraries, so as to calculate the processing of business logics such as superimposed graphic texts according to the first interface. The second interface pre-connected to the open graphics library may be OpenGL ES (OpenGL for Embeded System). OpenGL ES is a reduced subset of OpenGL and is an advanced 3D graphics API targeting handheld and embedded devices, which supports desktop systems and multiple platforms. By using the API interface of IMGUI and OpenGL ES, framebuffer (i.e., frame buffer) can be generated. Hereinafter, framebuffer will be abbreviated as fb.

[0069] Step S22: Obtain the frame buffer data by using a preset third interface, and use a preset general buffer manager to convert the frame buffer data into frame buffer data with a preset identifier; the preset identifier is an identifier recognizable by the direct rendering manager.

[0070] In a specific embodiment, the third interface may be EGL. EGL can mask the differences on different platforms. By swapping the buffer, it can obtain the framebuffer after the OpenGL rendering ends. Then, through the abstraction layer provided by GBM (i.e., Generic Buffer Manager, general buffer manager), it requests the buffer of the underlying memory management system of the platform to obtain the buffer, and then converts it into a drm fb with a preset identifier fb_id that can be used by the direct rendering manager. It should be noted that in this embodiment, GBM is used to better implement the docking with DRM.

[0071] Step S23: Use the frame buffer data with the preset identifier as the current user interface rendering data.

[0072] In this embodiment, the frame buffer data (drm fb) with the preset identifier can be used as the current user interface rendering data. Then, the drm fb is sent to the direct rendering manager.

[0073] Such as Figure 4 As a schematic diagram of the working processes of a UI thread, a cursor thread, and a video thread, after the ui-thread (i.e., the UI thread) generates UI data through the processes of steps S11 - S13, it updates the UI data to the render thread in the direct rendering manager (i.e., the render-thread in the figure); after the cursor-thread (i.e., the cursor thread) generates cursor rendering data, it updates the cursor rendering data to the render thread in the direct rendering manager; after the vidio-thread (i.e., the video thread) generates video rendering data through the relevant processes of hardware decoding the video stream, it updates the video rendering data to the render thread (i.e., the render-thread in the figure) in the direct rendering manager. The render thread in the direct rendering manager will check whether the UI data, cursor data, and video data have changed. If they have changed, it will perform rendering and screen display on the pre-connected display according to a preset screen update period and using these rendering data.

[0074] Figure 5A schematic diagram of data rendering proposed in this application is shown in the figure. The figure shows the process of using DRM to uniformly manage three types of rendering data, namely cursor, UI, and video, and rendering these three types of data onto the display screen. The generation process of UI data in the figure is expanded, that is, UI data is generated using the UI API interface, OpenGL, EGL, and GBM.

[0075] In this embodiment, the process of determining the user interface rendering data is elaborated in detail. That is, the UI data is determined by using the UI API interface, OpenGL, EGL, and GBM. This avoids the need to use the relatively large and bloated SDL graphics library in the prior art and streamline some modules. After deleting some unnecessary redundant functions, the performance of the linux cloud game client is improved with a more streamlined module. Moreover, the existing open-source linux UI framework does not include the GBM module. This solution adds the GBM module to better interface with the DRM of the video, improving the high-performance optimization of the cloud game streaming client rendering and promoting the development of the cloud game system.

[0076] See Figure 6 As shown, an embodiment of this application discloses a rendering-on-screen device, which may specifically include:

[0077] A rendering data determination module 11, configured to determine the current rendering data and send the rendering data to the direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data;

[0078] An on-screen data determination module 12, configured to determine target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy;

[0079] An on-screen display module 13, configured to merge the target on-screen data with the current layer and send the merged layer to a preset display screen for display.

[0080] In this application, first, the current rendering data is determined and sent to the direct rendering manager in the client system. The rendering data includes user interface rendering data, cursor rendering data, and video rendering data. Then, through the direct rendering manager and using a preset screen display optimization strategy, the target screen display data is determined from the rendering data. Next, the target screen display data is merged with the current layer, and the merged layer is sent to a preset display screen for display. In this way, after determining the current rendering data, this solution sends the rendering data to the direct rendering manager in the client system so that the direct rendering manager can uniformly manage this rendering data, and the streamlined rendering process avoids the screen display conflicts in the underlying framework in the prior art. In addition, this solution determines the target screen display data based on a preset screen display optimization strategy, which can ensure that the latest video rendering data is used for display each time the screen is displayed, thereby completing the optimization of the screen display.

[0081] In some specific embodiments, in the rendering data determination module 11, it includes:

[0082] The first frame buffer data generation unit is used to generate frame buffer data by using a first interface pre-connected to the graphical user interface and a second interface pre-connected to the Open Graphics Library.

[0083] The second frame buffer data generation unit is used to obtain the frame buffer data by using a preset third interface and convert the frame buffer data into frame buffer data with a preset identifier by using a preset general buffer manager; the preset identifier is an identifier recognizable by the direct rendering manager.

[0084] The user interface rendering data determination unit is used to use the frame buffer data with the preset identifier as the current user interface rendering data.

[0085] In some specific embodiments, in the rendering data determination module 11, it includes:

[0086] The pixel data acquisition unit is used to acquire the cursor pixel data sent by the host side; the cursor pixel data is data representing the pixel values of the cursor image.

[0087] The cursor rendering data determination unit is used to determine the cursor rendering data based on the rendering pixel data and by using a preset cursor thread; the cursor rendering data includes the position coordinate data of the cursor, the resolution data of the cursor, and the color mode data of the cursor.

[0088] In some specific embodiments, in the rendering data determination module 11, it includes:

[0089] The video bitstream acquisition unit is used to acquire the video bitstream sent by the host side.

[0090] A video decoding unit for performing hardware decoding on the video bitstream to obtain video data corresponding to the video bitstream and using the video data as video rendering data.

[0091] In some specific embodiments, the rendering and screen display device further includes:

[0092] A data storage unit for storing the video rendering data in a preset stack when the direct rendering manager receives the video rendering data;

[0093] A video screen display data determination unit for determining target screen display data from all the video rendering data in the preset stack.

[0094] In some specific embodiments, the rendering and screen display device further includes:

[0095] A quantity determination unit for determining the quantity of the video rendering data in the stack;

[0096] A latest data determination unit for, if the quantity of the video rendering data in the stack is greater than 1, determining the latest video rendering data from all the video rendering data and popping out the video rendering data other than the latest video rendering data from all the video rendering data.

[0097] In some specific embodiments, the rendering and screen display device further includes:

[0098] A period acquisition unit for acquiring a screen display period through a preset time acquisition interface;

[0099] Correspondingly, the screen display module 13 includes:

[0100] A period application unit for merging the target screen display data with the current layer based on the screen display period and sending the merged layer to a preset display screen for display.

[0101] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 7 which is a structural diagram of the electronic device 20 shown according to an exemplary embodiment, and the content in the figure should not be considered as any limitation to the scope of use of the present application.

[0102] Figure 7Schematic diagram of the structure of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a display screen 24, an input / output interface 25, a communication interface 26, and a communication bus 27. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the rendering and screen display method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 26 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0104] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be short-term storage or permanent storage.

[0105] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the rendering and screen display method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0106] Furthermore, the present application also discloses a computer-readable storage medium. The computer-readable storage medium mentioned here includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, or optical discs, or any other form of storage medium well-known in the technical field. Among them, when the computer program is executed by the processor, it implements the rendering and screen display method disclosed above. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0107] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section. Professionals can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0108] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0109] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0110] The above has introduced in detail the rendering and screen display method, device, equipment, and storage medium provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A rendering and display method, characterized in that, Applied to a client system, including: Determine the current rendering data and send the rendering data to the direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data; Determine target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy; Merge the target on-screen data with the current layer and send the merged layer to a preset display screen for display; Among them, the determining of the current rendering data includes: obtaining cursor pixel data sent by the host side; the cursor pixel data is data representing the pixel values of the cursor image; based on the cursor pixel data, use a preset cursor thread to determine cursor rendering data; the cursor rendering data includes the position coordinate data of the cursor, the resolution data of the cursor, and the color mode data of the cursor; The determining of the target on-screen data from the rendering data through the direct rendering manager and using a preset on-screen optimization strategy further includes: when the direct rendering manager receives the video rendering data, store the video rendering data in a preset stack; determine the target on-screen data from all the video rendering data in the preset stack; The rendering on-screen method further includes: determining the data quantity of the video rendering data in the stack; if the data quantity of the video rendering data in the stack is greater than 1, determine the latest video rendering data from all the video rendering data and pop the video rendering data other than the latest video rendering data from all the video rendering data; The rendering on-screen method further includes: obtaining the on-screen cycle through a preset time acquisition interface; correspondingly, the merging of the target on-screen data with the current layer and sending the merged layer to a preset display screen for display includes: merging the target on-screen data with the current layer based on the on-screen cycle and sending the merged layer to a preset display screen for display.

2. The rendering and display method according to claim 1, wherein The determining of the current rendering data includes: Generate frame buffer data using a first interface pre-connected to the graphical user interface and a second interface pre-connected to the open graphics library; Obtain the frame buffer data using a preset third interface and use a preset general buffer manager to convert the frame buffer data into frame buffer data with a preset identifier; the preset identifier is an identifier recognizable by the direct rendering manager; Use the frame buffer data with the preset identifier as the current user interface rendering data.

3. The rendering and display method according to claim 1, wherein The determining of the current rendering data includes: Obtain the video bitstream sent by the host side; Perform hardware decoding on the video bitstream to obtain video data corresponding to the video bitstream and use the video data as video rendering data.

4. A rendering and screen display device, characterized in that, Applied to a client system, including: A rendering data determination module, configured to determine the current rendering data and send the rendering data to the direct rendering manager in the client system; the rendering data includes user interface rendering data, cursor rendering data, and video rendering data; The upper screen data determination module is configured to determine target upper screen data from the rendering data through the direct rendering manager and by using a preset upper screen optimization strategy; The upper screen display module is configured to merge the target upper screen data with the current layer and send the merged layer to a preset display screen for display; Among them, the rendering data determination module includes: a pixel data acquisition unit configured to acquire cursor pixel data sent by the host; the cursor pixel data is data representing the pixel values of the cursor image; a cursor rendering data determination unit configured to determine cursor rendering data based on the cursor pixel data and by using a preset cursor thread; the cursor rendering data includes the position coordinate data of the cursor, the resolution data of the cursor, and the color mode data of the cursor; The rendering and upper screen device further includes: a data storage unit configured to store the video rendering data in a preset stack when the direct rendering manager receives the video rendering data; a video upper screen data determination unit configured to determine target upper screen data from all the video rendering data in the preset stack; The rendering and upper screen device further includes: a quantity determination unit configured to determine the data quantity of the video rendering data in the stack; a latest data determination unit configured to, if the data quantity of the video rendering data in the stack is greater than 1, determine the latest video rendering data from all the video rendering data and pop the video rendering data other than the latest video rendering data from all the video rendering data; The rendering and upper screen device further includes: a period acquisition unit configured to acquire the upper screen period through a preset time acquisition interface; correspondingly, the upper screen display module includes: a period application unit configured to merge the target upper screen data with the current layer based on the upper screen period and send the merged layer to a preset display screen for display.

5. An electronic device, characterized in that, It includes a processor and a memory; among them, when the processor executes the computer program stored in the memory, the rendering and upper screen method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that, It is used to store a computer program; among them, when the computer program is executed by a processor, the rendering and upper screen method according to any one of claims 1 to 3 is implemented.

Images