Rendering method, device, equipment, computer-readable storage medium, and program product

Abstract

The embodiments of the present application provide a rendering method, apparatus, device, computer-readable storage medium and program product, which relate to cloud computing, cloud games, maps and other fields, and the application scenarios include but are not limited to game scenarios. The method includes: obtaining the predicted graphics processor utilization corresponding to any frame to be rendered in the video; determining the complexity of each drawing call in multiple drawing calls corresponding to any frame to be rendered; based on the predicted graphics processor utilization and the complexity of each drawing call, determining the rendering method of each drawing call, the rendering method includes any one of client rendering and server rendering; based on the rendering method of each drawing call, rendering any frame to be rendered, and determining the rendered picture. In this way, the advantages of no delay in client rendering and the advantages of large computing power of server rendering are combined, saving the graphics processor resources corresponding to the client and improving the display quality of the rendered picture.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically, to a rendering method, device, equipment, computer-readable storage medium, and program product. Background Art

[0002] In the prior art, the game screen to be rendered is rendered by the client in the terminal to obtain the game screen, and the terminal displays the game screen; for example, all the game screens to be rendered in the game mall are rendered by the client in the terminal. If the terminal configuration is poor, the display quality of the game screen with large resource consumption is poor, that is, the rendering quality corresponding to the game screen is low; for example, the display quality of the game prop skin in the game screen is poor, that is, the rendering quality corresponding to the game prop skin is low, which leads to the user's lack of willingness to purchase the game prop skin. Summary of the invention

[0003] In view of the shortcomings of existing methods, the present application proposes a rendering method, device, equipment, computer-readable storage medium and computer program product to solve the problem of how to improve the rendering quality corresponding to the game screen.

[0004] In a first aspect, the present application provides a rendering method, comprising:

[0005] Obtain the predicted GPU utilization corresponding to any frame to be rendered in the video;

[0006] Determine the complexity of each of the multiple draw calls corresponding to any frame of the to-be-rendered screen;

[0007] Determine a rendering mode for each draw call based on the predicted GPU utilization and the complexity of each draw call, where the rendering mode includes either client rendering or server rendering;

[0008] Based on the rendering mode of each drawing call, any frame to be rendered is rendered to determine the rendered picture.

[0009] In one embodiment, determining the complexity of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture includes:

[0010] Determine the rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture;

[0011] The complexity of each draw call is determined based on the rendering type of each draw call and the graphics application program interface of each draw call.

[0012] In one embodiment, determining the rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture includes:

[0013] Use the preset analysis tool to analyze the video offline to obtain the rendering state characteristics of the draw call of any rendering type;

[0014] If the rendering state characteristics of each draw call are the same as the rendering state characteristics of a draw call of any rendering type, the rendering type of each draw call is determined to be any rendering type; the rendering state characteristics of each draw call include the graphics application program interface characteristics of each draw call and the shader characteristics of each draw call.

[0015] In one embodiment, the complexity of each draw call is determined based on the rendering type of each draw call and the graphics application program interface of each draw call, including:

[0016] Based on the rendering type of each draw call, determine the initial complexity of each draw call, where the initial complexity of each draw call is used to characterize the preliminary predicted rendering cost of each draw call;

[0017] Determining, based on a graphics application programming interface for each draw call, the number of models per draw call, the model complexity per draw call, and the number of shader instructions per draw call;

[0018] The complexity of each draw call is determined based on the initial complexity of each draw call, the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call, and the complexity of each draw call is used to characterize the predicted rendering cost of each draw call.

[0019] In one embodiment, determining the number of models for each draw call, the model complexity for each draw call, and the number of shader instructions for each draw call based on a graphics application program interface for each draw call includes:

[0020] Determining the number of models per draw call and the model complexity per draw call through a graphics application program interface for each draw call;

[0021] Determine the number of shader instructions per draw call through the graphics API's reflection mechanism.

[0022] In one embodiment, based on the predicted GPU utilization and the complexity of each draw call, determining the rendering mode of each draw call includes:

[0023] A rendering mode for each draw call is determined based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold; the preset first utilization threshold is greater than the preset second utilization threshold, and the preset first complexity threshold is greater than the preset second complexity threshold.

[0024] In one embodiment, determining a rendering mode for each draw call based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold includes:

[0025] If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each drawing call is greater than or equal to the first complexity threshold, determining the rendering mode of each drawing call to be the server-side rendering;

[0026] If the predicted GPU utilization is less than the first utilization threshold, and the complexity of each draw call is less than the first complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0027] If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0028] If the predicted GPU utilization is greater than or equal to a first utilization threshold, the complexity of each draw call is less than a first complexity threshold, and the complexity of each draw call is less than a second complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0029] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to a second utilization threshold, determining that the rendering mode of each draw call is server-side rendering;

[0030] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than a second utilization threshold, then the rendering mode of each draw call is determined to be client rendering.

[0031] In one embodiment, based on the rendering mode of each drawing call, any frame of a to-be-rendered picture is rendered, and a rendered picture is determined, including:

[0032] If the rendering mode of a part of the draw calls in each draw call is client rendering, and the rendering mode of another part of the draw calls in each draw call is server rendering, then the part of the draw calls is rendered to obtain client rendering results, and the server rendering results for the other part of the draw calls sent by the server are received;

[0033] The client rendering results and the server rendering results are mixed and superimposed to determine the rendered image.

[0034] In a second aspect, the present application provides a rendering device, including:

[0035] The first processing module is used to obtain the predicted graphics processor utilization corresponding to any frame to be rendered in the video;

[0036] The second processing module is used to determine the complexity of each drawing call among multiple drawing calls corresponding to any frame of the to-be-rendered picture;

[0037] A third processing module is used to determine a rendering mode for each drawing call based on the predicted GPU utilization and the complexity of each drawing call, where the rendering mode includes any one of client rendering and server rendering;

[0038] The fourth processing module is used to render any frame of the to-be-rendered picture based on the rendering mode of each drawing call, and determine the rendered picture.

[0039] In a third aspect, the present application provides an electronic device, including: a processor, a memory, and a bus;

[0040] A bus, used to connect the processor and memory;

[0041] A memory, used for storing operation instructions;

[0042] The processor is used to execute the rendering method of the first aspect of the present application by calling an operation instruction.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, and the computer program is used to execute the rendering method of the first aspect of the present application.

[0044] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the rendering method in the first aspect of the present application when the computer program is executed by a processor.

[0045] The technical solution provided in the embodiments of the present application has at least the following beneficial effects:

[0046] Obtain the predicted GPU utilization rate corresponding to any frame to be rendered in the video; determine the complexity of each drawing call in multiple drawing calls corresponding to any frame to be rendered; determine the rendering method of each drawing call based on the predicted GPU utilization rate and the complexity of each drawing call, the rendering method includes any one of client rendering and server rendering; based on the rendering method of each drawing call, render any frame to be rendered to determine the rendered image. In this way, by determining the rendering method of each drawing call in multiple drawing calls corresponding to any frame to be rendered, the advantages of no delay in client rendering to make the video smooth and the advantages of large computing power of server rendering can be combined, that is, based on the rendering method of each drawing call, any frame to be rendered is rendered to obtain the rendered image, which saves the corresponding GPU resources of the client and improves the display quality of the rendered image, that is, improves the rendering quality corresponding to the rendered image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0048] Figure 1 A schematic diagram of the architecture of a rendering system provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a rendering method provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of a rendering method provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a rendering provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of a rendering device provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0055] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" indicates that it is implemented as "A", or implemented as "B", or implemented as "A and B".

[0056] It can be understood that in the specific implementation of this application, rendering-related data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0058] An embodiment of the present application is a rendering method provided by a rendering system, which relates to cloud computing, cloud games, maps and other fields.

[0059] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0060] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0061] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively and comprehensively applies advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0062] In order to better understand and illustrate the solutions of the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below.

[0063] DirectX (Direct eXtension, DX for short): DirectX is an application programming interface (API) that allows games or multimedia programs based on the Windows platform to achieve higher execution efficiency.

[0064] OpenGL: OpenGL is used to draw simple graphics bits to complex three-dimensional scenes; OpenGL is commonly used in CAD, virtual reality, scientific visualization programs and video game development.

[0065] Rendering environment: The rendering environment can be a rendering context (Context); OpenGL's Context records all the information required for OpenGL rendering. The rendering environment is a large structure that records the color currently used for drawing, whether there is lighting calculation, and the light source that is turned on, etc. The status and status attributes set by OpenGL function calls.

[0066] Rendering resources: Rendering resources can be the basic data required for drawing, such as the triangle mesh of the model, skin map material, etc.

[0067] Rendering state: The rendering state is used to inform the GPU (Graphics Processing Unit) how to draw the picture; for example, to inform the GPU which shader, texture, material, light, etc. to use.

[0068] Drawcall: Each time the CPU (Central Processing Unit) prepares data and notifies the GPU (for example, the CPU calls the graphics API (Application Program Interface) and instructs the GPU to draw graphics), it is a draw call.

[0069] The solution provided by the embodiment of the present application relates to cloud computing technology. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] In order to better understand the solution provided by the embodiment of the present application, the solution is described below in conjunction with a specific application scenario.

[0071] In one embodiment, Figure 1 FIG. 1 shows a schematic diagram of the architecture of a rendering system applicable to an embodiment of the present application. It can be understood that the rendering method provided in the embodiment of the present application can be applicable to but not limited to applications such as Figure 1 In the application scenario shown.

[0072] In this example, Figure 1As shown, the architecture of the rendering system in this example may include but is not limited to a server 10, a terminal 20 and a database 30. The server 10, the terminal 20 and the database 30 may interact with each other through a network 40. The client in the terminal 20 obtains the predicted GPU utilization corresponding to any frame of the to-be-rendered picture in the video; the client in the terminal 20 determines the complexity of each drawing call in multiple drawing calls corresponding to any frame of the to-be-rendered picture; the client in the terminal 20 determines the rendering mode of each drawing call based on the predicted GPU utilization and the complexity of each drawing call, and the rendering mode includes any one of client rendering and server rendering; the client in the terminal 20 renders any frame of the to-be-rendered picture based on the rendering mode of each drawing call to determine the rendered picture. If the rendering mode of a part of the drawing calls in each drawing call is client rendering, and the rendering mode of another part of the drawing calls in each drawing call is server rendering, the client in the terminal 20 renders a part of the drawing calls to obtain the client rendering result, and receives the server rendering result for the other part of the drawing calls sent by the server 10 through the network 40; the terminal 20 determines the rendered picture by mixing and overlaying the client rendering result and the server rendering result. The terminal 20 saves the client rendering result, the server rendering result and the rendered picture in the database 30. The server is the server 10, and the server rendering result is the server rendering result.

[0073] It is understandable that the above is only an example and this embodiment is not limited here.

[0074] Among them, terminals include but are not limited to smart phones (such as Android phones, iOS phones, etc.), mobile phone simulators, tablet computers, laptops, digital broadcast receivers, MIDs (Mobile Internet Devices), PDAs (Personal Digital Assistants), intelligent voice interaction devices, smart home appliances, car terminals, etc.; among them, smart home appliances include smart TVs.

[0075] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0076] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0077] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0078] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0079] The so-called artificial intelligence cloud service is generally also called AIaaS (AI as a Service). This is the service mode of a mainstream artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access and use one or more artificial intelligence services provided by the platform through the API interface. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0080] The above network may include but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, Wi-Fi and other networks that realize wireless communication. The specific can also be determined based on the actual application scenario requirements and is not limited here.

[0081] See also Figure 2 , Figure 2 A flowchart of a rendering method provided by an embodiment of the present application is shown, wherein the method can be executed by any electronic device, such as a terminal, etc.; as an optional implementation, the method can be executed by a client in a terminal. For the convenience of description, in the description of some optional embodiments below, the client in a terminal will be used as an example to illustrate the execution subject of the method. Figure 2 As shown, the rendering method provided in the embodiment of the present application includes the following steps:

[0082] S201, obtaining a predicted GPU utilization rate corresponding to any frame to be rendered in a video.

[0083] Specifically, the video may be a game video, any frame to be rendered may be a frame to be rendered in the game, the graphics processor may be a GPU, and the graphics processor utilization may be a GPU utilization.

[0084] The GPU utilization rate corresponding to the previous frame to be rendered of any frame to be rendered is determined as the predicted GPU utilization rate corresponding to any frame to be rendered. For example, if any frame to be rendered is the Nth frame to be rendered, and the previous frame to be rendered is the N-1th frame to be rendered, the GPU utilization rate corresponding to the N-1th frame to be rendered can be used as the predicted GPU utilization rate corresponding to the Nth frame to be rendered; wherein the GPU utilization rate corresponding to the N-1th frame to be rendered is the actual GPU utilization rate, and N is a positive integer greater than 1.

[0085] S202, determining the complexity of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture.

[0086] Specifically, the client intercepts and caches the graphics application program interface and shader of each draw call in multiple draw calls corresponding to any frame to be rendered, generates a rendering instruction stream, and caches the rendering instruction stream; wherein the rendering instruction stream includes multiple draw calls, and a draw call in the multiple draw calls includes the rendering state of the draw call and the rendering resources of the draw call.

[0087] The complexity of each draw call is used to characterize the estimated rendering cost of each draw call.

[0088] S203, determining a rendering mode for each drawing call based on the predicted GPU utilization and the complexity of each drawing call, where the rendering mode includes any one of client-side rendering and server-side rendering.

[0089] Specifically, client rendering may be rendering of a certain drawing call by a client, and server rendering may be rendering of a certain drawing call by a server, wherein the server may be a server.

[0090] If the rendering mode of all draw calls in multiple draw calls is client rendering, any frame to be rendered is rendered by the client; if the rendering mode of all draw calls in multiple draw calls is server rendering, any frame to be rendered is rendered by the server; if the rendering mode of some draw calls in multiple draw calls is client rendering, and the rendering mode of another part of the draw calls in multiple draw calls is server rendering, the one part of the draw calls is rendered by the client, and the other part of the draw calls is rendered by the server.

[0091] S204, based on the rendering mode of each drawing call, rendering any frame of the to-be-rendered picture to determine the rendered picture.

[0092] Specifically, if the rendering modes of all the drawing calls in the multiple drawing calls are client rendering, any frame of the to-be-rendered picture is rendered by the client to obtain a rendered picture.

[0093] If the rendering mode of all the drawing calls in the multiple drawing calls is server rendering, then any frame to be rendered is rendered by the server to obtain a rendered picture.

[0094] If the rendering mode of a part of the multiple drawing calls is client rendering, and the rendering mode of another part of the multiple drawing calls is server rendering, then the part of the drawing calls is rendered by the client to obtain a client rendering result, and the other part of the drawing calls is rendered by the server to obtain a server rendering result. The client obtains a rendered picture by mixing and overlaying the client rendering result and the server rendering result.

[0095] In an embodiment of the present application, the predicted GPU utilization rate corresponding to any frame of the to-be-rendered screen in the video is obtained; the complexity of each drawing call in the multiple drawing calls corresponding to any frame of the to-be-rendered screen is determined; based on the predicted GPU utilization rate and the complexity of each drawing call, the rendering mode of each drawing call is determined, and the rendering mode includes any one of client rendering and server rendering; based on the rendering mode of each drawing call, any frame of the to-be-rendered screen is rendered to determine the rendered screen. In this way, by determining the rendering mode of each drawing call in the multiple drawing calls corresponding to any frame of the to-be-rendered screen, the advantage of the client rendering without delay to ensure high video fluency and the advantage of the server rendering computing power can be combined, that is, based on the rendering mode of each drawing call, any frame of the to-be-rendered screen is rendered to obtain the rendered screen, which saves the corresponding GPU resources of the client and improves the display quality of the rendered screen, that is, improves the rendering quality corresponding to the rendered screen.

[0096] In one embodiment, determining the complexity of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture includes:

[0097] Determine the rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture;

[0098] The complexity of each draw call is determined based on the rendering type of each draw call and the graphics application program interface of each draw call.

[0099] Specifically, the rendering type of the draw call may be PBR (Physically Based Rendering), particle special effect rendering, UI (User Interface) rendering, vegetation rendering, etc. Each draw call may correspond to a rendering type. The graphics application program interface of each draw call may include an upload model resource API, a drawing API, a display API, etc.

[0100] In one embodiment, determining the rendering type of each of the multiple drawing calls corresponding to any frame to be rendered includes steps A1-A2:

[0101] Step A1: Perform offline analysis on the video using a preset analysis tool to obtain rendering state features of a draw call of any rendering type.

[0102] Specifically, the preset analysis tool may be a frame capture tool such as renderdoc; through the preset analysis tool, some frames in the video may be analyzed offline to obtain the rendering state characteristics of a draw call of any rendering type. The rendering state characteristics of a draw call may include the graphics application program interface characteristics (graphics API characteristics) of the draw call and the shader characteristics (shader characteristics) of the draw call.

[0103] For example, any rendering type is vegetation rendering, and the graphics API feature of a certain drawing call corresponding to the vegetation rendering is the drawing interface DrawIndexedInstanced (42, 947); the shader feature of the drawing call is the size of Vertex shader 94490, the md5 value of Vertex shader 94490, and the like.

[0104] Step A2: if the rendering state characteristics of each draw call are the same as the rendering state characteristics of a draw call of any rendering type, then the rendering type of each draw call is determined to be any rendering type; the rendering state characteristics of each draw call include the graphics application program interface characteristics of each draw call and the shader characteristics of each draw call.

[0105] Specifically, for example, if any rendering type is vegetation rendering, when the game is running, if the rendering state characteristics of a drawing call A among multiple drawing calls corresponding to the picture to be rendered in any frame are the same as the rendering state characteristics of the drawing call for vegetation rendering, then the rendering type of the drawing call A is determined to be vegetation rendering; wherein, the graphics application program interface characteristics of the drawing call A are the same as the graphics application program interface characteristics of the drawing call for vegetation rendering, and the shader characteristics of the drawing call A are the same as the shader characteristics of the drawing call for vegetation rendering.

[0106] In one embodiment, determining the complexity of each draw call based on a rendering type of each draw call and a graphics application program interface of each draw call comprises the steps of:

[0107] Based on the rendering type of each draw call, determine the initial complexity of each draw call, where the initial complexity of each draw call is used to characterize the preliminary predicted rendering cost of each draw call;

[0108] Determining, based on a graphics application programming interface for each draw call, the number of models per draw call, the model complexity per draw call, and the number of shader instructions per draw call;

[0109] The complexity of each draw call is determined based on the initial complexity of each draw call, the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call, and the complexity of each draw call is used to characterize the predicted rendering cost of each draw call.

[0110] Specifically, for example, the rendering type of a certain draw call B is vegetation rendering. Based on the rendering type of the draw call B, which is vegetation rendering, the initial complexity of the draw call B is determined.

[0111] The number of models of draw call B and the model complexity of draw call B are determined through the graphics application program interface of draw call B, wherein the graphics application program interface feature of draw call B is DrawIndexedInstanced(42, 947), the number of models of draw call B is InstanceCount 947, and the model complexity of draw call B is IndexCountPerInstance 42.

[0112] The number of shader instructions bound to draw call B is determined through the reflection mechanism of the graphics application program interface of draw call B; the reflection mechanism of the graphics application program interface of draw call B is implemented by the following function: ID3D11ShaderReflection::GetBitwiseInstructionCountmethod(d3d11shader.h); wherein the number of shader instructions bound to draw call B is UINTGetBitwiseInstructionCount().

[0113] The complexity of draw call B is determined based on the initial complexity of draw call B, the number of models of draw call B, the model complexity of draw call B, and the number of shader instructions of draw call B. The complexity of draw call B is used to characterize the predicted rendering cost of draw call B. The larger the initial complexity of draw call B, the number of models of draw call B, the model complexity of draw call B, and the number of shader instructions of draw call B, the greater the complexity of draw call B, that is, the greater the time consumption of draw call B; thus, the time consumption of draw call B can be identified, that is, the cost of vegetation rendering when the game is running can be estimated.

[0114] In one embodiment, based on the graphics application program interface of each draw call, determining the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call comprises steps B1-B2:

[0115] Step B1, determining the number of models for each draw call and the model complexity of each draw call through the graphics application program interface of each draw call.

[0116] For example, through the graphics application program interface of draw call B, the model quantity of draw call B and the model complexity of draw call B are determined, wherein the graphics application program interface feature of draw call B is DrawIndexedInstanced(42, 947), the model quantity of draw call B is InstanceCount 947, and the model complexity of draw call B is IndexCountPerInstance 42.

[0117] Step B2, determining the number of shader instructions for each drawing call through the reflection mechanism of the graphics application program interface.

[0118] For example, the number of shader instructions bound to draw call B is determined through the reflection mechanism of the graphics application program interface of draw call B; the reflection mechanism of the graphics application program interface of draw call B is implemented by the following function: ID3D11ShaderReflection::GetBitwiseInstructionCountmethod(d3d11shader.h); wherein the number of shader instructions bound to draw call B is UINTGetBitwiseInstructionCount().

[0119] In one embodiment, based on the predicted GPU utilization and the complexity of each draw call, determining the rendering mode of each draw call includes:

[0120] A rendering mode for each draw call is determined based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold; the preset first utilization threshold is greater than the preset second utilization threshold, and the preset first complexity threshold is greater than the preset second complexity threshold.

[0121] Specifically, for example, the second utilization threshold may be 80% of the first utilization threshold, and the second complexity threshold may be 85% of the first complexity threshold. The first utilization threshold may be the maximum utilization of the GPU.

[0122] In one embodiment, determining a rendering mode for each draw call based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold includes:

[0123] If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each draw call is greater than or equal to the first complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0124] If the predicted GPU utilization is less than the first utilization threshold, and the complexity of each draw call is less than the first complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0125] If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0126] If the predicted GPU utilization is greater than or equal to a first utilization threshold, the complexity of each draw call is less than a first complexity threshold, and the complexity of each draw call is less than a second complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0127] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to a second utilization threshold, determining that the rendering mode of each draw call is server-side rendering;

[0128] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than a second utilization threshold, then the rendering mode of each draw call is determined to be client rendering.

[0129] Specifically, the graphics processor utilization may be a GPU utilization, and the server may be a cloud server.

[0130] For example, if the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of the draw call C is greater than or equal to the first complexity threshold, then the rendering mode of the draw call C is determined to be server rendering; the client rendering capability is limited, and the draw call C can be sent to the server for rendering; in this way, the GPU resources corresponding to the client, that is, the local GPU resources, can be saved, and the server rendering computing power advantage can be brought into play.

[0131] For example, if the predicted GPU utilization is less than the first utilization threshold, and the complexity of the draw call C is less than the first complexity threshold, then the rendering mode of the draw call C is determined to be client rendering; the client rendering capability can meet the requirements for rendering the draw call C, that is, the draw call C is rendered by the client; in this way, the client rendering has no delay, and the video fluency can be high.

[0132] For example, if the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of the draw call C is less than the first complexity threshold, and the complexity of the draw call C is greater than or equal to the second complexity threshold, then the rendering mode of the draw call C is determined to be server-side rendering, wherein the second complexity threshold may be 85% of the first complexity threshold; the client rendering capability is limited, and the draw call C may be sent to the server for rendering; in this way, the GPU resources corresponding to the client, i.e., the local GPU resources, may be saved, and the server rendering computing power advantage may be brought into play.

[0133] For example, if the predicted GPU utilization is greater than or equal to a first utilization threshold, the complexity of the GPU utilization is less than the first complexity threshold, and the complexity of the GPU utilization is less than the second complexity threshold, then the rendering method of the GPU utilization is determined to be client rendering, wherein the second complexity threshold may be 85% of the first complexity threshold; the client rendering capability may satisfy the rendering of the drawing call C, that is, the drawing call C is rendered through the client; in this way, the client rendering has no delay, and the video fluency may be high.

[0134] For example, if the predicted GPU utilization is less than the first utilization threshold, the complexity of the draw call C is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to the second utilization threshold, then the rendering mode of the draw call C is determined to be server-side rendering, wherein the second utilization threshold may be 80% of the first utilization threshold; the client rendering capability is limited, and the draw call C may be sent to the server for rendering; in this way, the GPU resources corresponding to the client, i.e., the local GPU resources, may be saved, and the server rendering computing power advantage may be brought into play.

[0135] For example, if the predicted GPU utilization is less than a first utilization threshold, the complexity of the draw call C is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than a second utilization threshold, then the rendering mode of the draw call C is determined to be client rendering, wherein the second utilization threshold may be 80% of the first utilization threshold; the client rendering capability may satisfy the rendering of the draw call C, that is, the draw call C is rendered by the client; in this way, the client rendering has no delay, and the video fluency can be high.

[0136] In one embodiment, based on the rendering mode of each drawing call, any frame of a to-be-rendered picture is rendered, and a rendered picture is determined, including:

[0137] If the rendering mode of a part of the draw calls in each draw call is client rendering, and the rendering mode of another part of the draw calls in each draw call is server rendering, then the part of the draw calls is rendered to obtain client rendering results, and the server rendering results for the other part of the draw calls sent by the server are received;

[0138] The client rendering results and the server rendering results are mixed and superimposed to determine the rendered image.

[0139] Specifically, for example, the multiple drawing calls are 10 drawing calls. If the rendering method of 3 drawing calls among the 10 drawing calls is client rendering, and the rendering method of the other 7 drawing calls among the 10 drawing calls is server rendering, then the 3 drawing calls are rendered by the client to obtain client rendering results, and the 7 drawing calls are rendered by the server to obtain server rendering results; the server sends the rendering results to the client, and the client performs mixed overlay processing on the client rendering results and the server rendering results to obtain a rendered image. The mixed overlay processing can be implemented by alpha blending technology, that is, the client rendering results and the server rendering results are superimposed together to obtain a rendered image.

[0140] The application of the embodiments of the present application has at least the following beneficial effects:

[0141] The client determines the rendering mode of each of the multiple drawing calls corresponding to any frame of the to-be-rendered screen, so that the advantage of the client's rendering without delay to ensure high video fluency can be combined with the advantage of the server's large rendering computing power. That is, based on the rendering mode of each drawing call, any frame of the to-be-rendered screen is rendered to obtain the rendered screen, which saves the corresponding graphics processor resources of the client and improves the display quality of the rendered screen, that is, improves the rendering quality corresponding to the rendered screen.

[0142] In order to better understand the method provided in the embodiment of the present application, the scheme of the embodiment of the present application is further explained below with reference to examples of specific application scenarios.

[0143] In a specific application scenario embodiment, such as a game scenario, see Figure 3 , shows a processing flow of a rendering method, such as Figure 3 As shown, the processing flow of the rendering method provided in the embodiment of the present application includes the following steps:

[0144] S301, the client in the terminal generates and caches a rendering instruction stream corresponding to the Nth frame of the game screen to be rendered in the game video, and the client sends the rendering status and rendering resources of all drawing calls in the rendering instruction stream to the server.

[0145] Specifically, the client intercepts and caches the graphics application program interface and shader of each drawing call in the multiple drawing calls corresponding to the Nth frame of the game screen to be rendered, generates a rendering instruction stream, and caches the rendering instruction stream; wherein the rendering instruction stream includes multiple drawing calls, and a drawing call in the multiple drawing calls includes the rendering state of the drawing call and the rendering resources of the drawing call, and N is a positive integer greater than 1.

[0146] For example, Figure 4As shown, the server is a cloud server; the client caches the rendering instructions corresponding to the Nth frame of the game screen to be rendered; the client sends the rendering status and rendering resources of all drawing calls in the rendering instructions to the cloud server through asynchronous transmission; among them, DX or OpenGL is a collection of tools for preparing rendering resources and rendering status for the GPU.

[0147] S302: The server receives the rendering status and rendering resources of all drawing calls sent by the client, and initializes and updates the rendering environment.

[0148] Specifically, for example, Figure 4 As shown, after the cloud server receives the rendering status and rendering resources sent by the client, the cloud server initializes and updates the rendering environment, wherein the rendering environment may include a rendering context and the like.

[0149] S303, the client obtains the GPU utilization and CPU utilization corresponding to the N-1th frame of the game screen to be rendered, and determines the GPU utilization corresponding to the N-1th frame of the game screen to be rendered as the predicted GPU utilization corresponding to the Nth frame of the game screen to be rendered.

[0150] Specifically, the GPU utilization corresponding to the N-1th frame of the game screen to be rendered is the actual GPU utilization, and the CPU utilization corresponding to the N-1th frame of the game screen to be rendered is the actual CPU utilization. The CPU utilization corresponding to the N-1th frame of the game screen to be rendered can be used as an option to participate in the subsequent determination of the rendering mode of each drawing call.

[0151] For example, Figure 4 As shown, the client obtains the GPU utilization and CPU utilization corresponding to the N-1th frame of the game screen to be rendered.

[0152] S304, the client determines the rendering type of each drawing call in all drawing calls corresponding to the Nth frame of the game screen to be rendered.

[0153] Specifically, for example, Figure 4 As shown, the graphics application program interface characteristics (graphics API characteristics) of each draw call and the shader characteristics (shader characteristics) of each draw call are dynamically analyzed to determine the rendering type of each draw call. The rendering type of the draw call can be vegetation rendering, PBR, particle special effects, UI rendering, etc.

[0154] Dynamically analyze the graphics application program interface characteristics (graphics API characteristics) of each draw call and the shader characteristics (shader characteristics) of each draw call to determine the rendering type of each draw call, including:

[0155] By using a preset analysis tool, the video is analyzed offline to obtain the rendering state characteristics of a drawing call of any rendering type; for example, the preset analysis tool may be a frame grabbing tool such as renderdoc; by using the preset analysis tool, some frames in the video may be analyzed offline to obtain the rendering state characteristics of a drawing call of any rendering type; the rendering state characteristics of a certain drawing call may include a graphics application program interface characteristic (graphics API characteristic) of the drawing call and a shader characteristic (shader characteristic) of the drawing call; for example, if any rendering type is vegetation rendering, the graphics API characteristic of a certain drawing call corresponding to vegetation rendering is a drawing interface DrawIndexedInstanced (42, 947); the shader characteristics of the drawing call are the size of Vertex shader 94490, the md5 value of Vertex shader 94490, etc.;

[0156] If the rendering state characteristics of each draw call are the same as the rendering state characteristics of a draw call of any rendering type, then the rendering type of each draw call is determined to be any rendering type; the rendering state characteristics of each draw call include the graphics application program interface characteristics of each draw call and the shader characteristics of each draw call; for example, if any one of the rendering types is vegetation rendering, when the game is running, if the rendering state characteristics of a draw call A among multiple draw calls corresponding to any frame of the to-be-rendered screen are the same as the rendering state characteristics of a draw call for vegetation rendering, then the rendering type of the draw call A is determined to be vegetation rendering; wherein, the graphics application program interface characteristics of the draw call A are the same as the graphics application program interface characteristics of the draw call for vegetation rendering, and the shader characteristics of the draw call A are the same as the shader characteristics of the draw call for vegetation rendering.

[0157] S305 : The client determines the complexity of each draw call based on the rendering type of each draw call and the graphics application program interface of each draw call.

[0158] Specifically, based on the rendering type of each draw call, the initial complexity of each draw call is determined, and the initial complexity of each draw call is used to characterize the preliminary predicted rendering overhead of each draw call; based on the graphics application program interface of each draw call, the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call are determined; based on the initial complexity of each draw call, the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call, the complexity of each draw call is determined, and the complexity of each draw call is used to characterize the predicted rendering overhead of each draw call.

[0159] For example, Figure 4As shown, the complexity of each draw call is identified.

[0160] S306: The client determines a rendering mode for each drawing call based on the predicted GPU utilization, the complexity of each drawing call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold.

[0161] For example, Figure 4 As shown, the client determines the rendering mode of each draw call based on the GPU utilization and the complexity of each draw call, thereby subdividing the rendering mode of each draw call in all draw calls; that is: the client determines the size relationship between the predicted GPU utilization and the utilization threshold, and the utilization threshold includes at least one of a preset first utilization threshold and a preset second utilization threshold; the client determines the size relationship between the complexity of each draw call and the complexity threshold, and the complexity threshold includes at least one of a preset first complexity threshold and a preset second complexity threshold.

[0162] Determining how each draw call is rendered involves:

[0163] If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each draw call is greater than or equal to the first complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0164] If the predicted GPU utilization is less than a first utilization threshold, and the complexity of each draw call is less than a first complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0165] If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0166] If the predicted GPU utilization is greater than or equal to a first utilization threshold, the complexity of each draw call is less than a first complexity threshold, and the complexity of each draw call is less than a second complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0167] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted graphics processor utilization is greater than or equal to a second utilization threshold, determining that the rendering mode of each draw call is server-side rendering;

[0168] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted graphics processor utilization is less than a second utilization threshold, then the rendering mode of each draw call is determined to be client rendering.

[0169] S307, based on the rendering methods of the drawing calls, the Nth frame of the game screen to be rendered is rendered to determine the rendered screen.

[0170] For example, Figure 4 As shown, if the rendering method of all drawing calls corresponding to the Nth frame of the game screen to be rendered is client rendering, then the Nth frame of the game screen to be rendered is rendered by the client, that is, the entire frame is rendered locally; the client renders the Nth frame of the game screen to be rendered (that is, executes the rendering instructions locally) to obtain the rendered picture; the client displays the rendered picture through the terminal.

[0171] If the rendering method of all drawing calls corresponding to the Nth frame of the game screen to be rendered is server-side rendering, then the Nth frame of the game screen to be rendered is rendered by the server, that is, the entire frame is rendered by the cloud server; the client transmits the rendering instruction stream to the cloud server, the cloud server receives the rendering instruction stream sent by the client, and the cloud server executes the rendering instruction to obtain the rendered picture, that is, the rendering result; the cloud server stores the rendering result through FBO (Frame Buffer Object). The cloud server compresses the rendering result into a video stream and transmits the video stream to the client; the client receives the video stream and decodes the video stream to obtain the rendering result, that is, the rendered picture. The client displays the rendered picture through the terminal.

[0172] If the rendering method of a part of all drawing calls corresponding to the game screen to be rendered in the Nth frame is client rendering, and the rendering method of another part of all drawing calls is server rendering, then the part of drawing calls is rendered by the client to obtain the client rendering result, and the other part of drawing calls is rendered by the cloud server to obtain the server rendering result. The client obtains the rendered picture by mixing and overlaying the client rendering result and the server rendering result, that is, the rendered picture is generated by overlaying. For example, the drawing call with the rendering type of UI rendering is rendered by the client to obtain the client rendering result; the drawing call with the rendering type of PBR rendering is rendered by the cloud server to obtain the server rendering result. The client generates the rendered picture by overlaying the client rendering result and the server rendering result. The client displays the rendered picture through the terminal.

[0173] It should be noted that any frame of the game screen to be rendered in the game video can be rendered according to steps S301-S307 to obtain the rendered picture. Cloud gaming services are provided for game scenes with high resource consumption, and cloud gaming services are not provided for other game scenes to ensure smoothness. For example, cloud gaming services are provided in the game mall, that is, rendering is performed through the cloud server to ensure that the picture has the highest special effects (this game scene has relatively high requirements on GPU performance); for game scenes in the game, in order to ensure no delay, the client is used for rendering.

[0174] The application of the embodiments of the present application has at least the following beneficial effects:

[0175] The client determines the rendering mode of each drawing call among all drawing calls corresponding to any frame of the game screen to be rendered in the game video, so as to combine the advantage of client rendering without delay to make the video smooth and the advantage of the cloud server's large rendering computing power. That is, based on the rendering mode of each drawing call, any frame of the game screen to be rendered is rendered to obtain the rendered picture, which saves the corresponding GPU resources of the client and improves the display quality of the rendered game screen, that is, improves the rendering quality corresponding to the rendered game screen.

[0176] The present application also provides a rendering device. The structural diagram of the rendering device is as follows: Figure 5 As shown, the rendering device 60 includes a first processing module 601 , a second processing module 602 , a third processing module 603 and a fourth processing module 604 .

[0177] The first processing module 601 is used to obtain the predicted GPU utilization rate corresponding to any frame to be rendered in the video;

[0178] The second processing module 602 is used to determine the complexity of each drawing call among multiple drawing calls corresponding to any frame to be rendered;

[0179] A third processing module 603 is used to determine a rendering mode for each drawing call based on the predicted GPU utilization and the complexity of each drawing call, where the rendering mode includes any one of client rendering and server rendering;

[0180] The fourth processing module 604 is used to render any frame of the to-be-rendered picture based on the rendering mode of each drawing call, and determine the rendered picture.

[0181] In one embodiment, the second processing module 602 is specifically configured to:

[0182] Determine the rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture;

[0183] The complexity of each draw call is determined based on the rendering type of each draw call and the graphics application program interface of each draw call.

[0184] In one embodiment, the second processing module 602 is specifically configured to:

[0185] Use the preset analysis tool to analyze the video offline to obtain the rendering state characteristics of the draw call of any rendering type;

[0186] If the rendering state characteristics of each draw call are the same as the rendering state characteristics of a draw call of any rendering type, the rendering type of each draw call is determined to be any rendering type; the rendering state characteristics of each draw call include the graphics application program interface characteristics of each draw call and the shader characteristics of each draw call.

[0187] In one embodiment, the second processing module 602 is specifically configured to:

[0188] Based on the rendering type of each draw call, determine the initial complexity of each draw call, where the initial complexity of each draw call is used to characterize the preliminary predicted rendering cost of each draw call;

[0189] Determining, based on a graphics application programming interface for each draw call, the number of models per draw call, the model complexity per draw call, and the number of shader instructions per draw call;

[0190] The complexity of each draw call is determined based on the initial complexity of each draw call, the number of models for each draw call, the model complexity of each draw call, and the number of shader instructions for each draw call, and the complexity of each draw call is used to characterize the predicted rendering cost of each draw call.

[0191] In one embodiment, the second processing module 602 is specifically configured to:

[0192] Determining the number of models per draw call and the model complexity per draw call through a graphics application program interface for each draw call;

[0193] Determine the number of shader instructions per draw call through the graphics API's reflection mechanism.

[0194] In one embodiment, the third processing module 603 is specifically configured to:

[0195] A rendering mode for each draw call is determined based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold; the preset first utilization threshold is greater than the preset second utilization threshold, and the preset first complexity threshold is greater than the preset second complexity threshold.

[0196] In one embodiment, determining a rendering mode for each draw call based on a predicted graphics processor utilization, a complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold includes:

[0197] If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each drawing call is greater than or equal to the first complexity threshold, determining the rendering mode of each drawing call to be the server-side rendering;

[0198] If the predicted GPU utilization is less than the first utilization threshold, and the complexity of each draw call is less than the first complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0199] If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is server-side rendering;

[0200] If the predicted GPU utilization is greater than or equal to a first utilization threshold, the complexity of each draw call is less than a first complexity threshold, and the complexity of each draw call is less than a second complexity threshold, determining that the rendering mode of each draw call is client rendering;

[0201] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to a second utilization threshold, determining that the rendering mode of each draw call is server-side rendering;

[0202] If the predicted GPU utilization is less than a first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than a second utilization threshold, then the rendering mode of each draw call is determined to be client rendering.

[0203] In one embodiment, the fourth processing module 604 is specifically configured to:

[0204] If the rendering mode of a part of the draw calls in each draw call is client rendering, and the rendering mode of another part of the draw calls in each draw call is server rendering, then the part of the draw calls is rendered to obtain client rendering results, and the server rendering results for the other part of the draw calls sent by the server are received;

[0205] The client rendering results and the server rendering results are mixed and superimposed to determine the rendered image.

[0206] The application of the embodiments of the present application has at least the following beneficial effects:

[0207] Obtain the predicted GPU utilization rate corresponding to any frame to be rendered in the video; determine the complexity of each drawing call in multiple drawing calls corresponding to any frame to be rendered; determine the rendering method of each drawing call based on the predicted GPU utilization rate and the complexity of each drawing call, the rendering method includes any one of client rendering and server rendering; based on the rendering method of each drawing call, render any frame to be rendered to determine the rendered image. In this way, by determining the rendering method of each drawing call in multiple drawing calls corresponding to any frame to be rendered, the advantages of no delay in client rendering to make the video smooth and the advantages of large computing power of server rendering can be combined, that is, based on the rendering method of each drawing call, any frame to be rendered is rendered to obtain the rendered image, which saves the corresponding GPU resources of the client and improves the display quality of the rendered image, that is, improves the rendering quality corresponding to the rendered image.

[0208] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 6 As shown, Figure 6 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0209] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0210] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0211] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0212] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0213] Among them, electronic equipment includes but is not limited to: terminals, etc.

[0214] The application of the embodiments of the present application has at least the following beneficial effects:

[0215] Obtain the predicted GPU utilization rate corresponding to any frame to be rendered in the video; determine the complexity of each drawing call in multiple drawing calls corresponding to any frame to be rendered; determine the rendering method of each drawing call based on the predicted GPU utilization rate and the complexity of each drawing call, the rendering method includes any one of client rendering and server rendering; based on the rendering method of each drawing call, render any frame to be rendered to determine the rendered image. In this way, by determining the rendering method of each drawing call in multiple drawing calls corresponding to any frame to be rendered, the advantages of no delay in client rendering to make the video smooth and the advantages of large computing power of server rendering can be combined, that is, based on the rendering method of each drawing call, any frame to be rendered is rendered to obtain the rendered image, which saves the corresponding GPU resources of the client and improves the display quality of the rendered image, that is, improves the rendering quality corresponding to the rendered image.

[0216] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0217] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0218] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in any optional embodiment of the present application.

[0219] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0220] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A rendering method, characterized in that: include: Obtain the predicted GPU utilization corresponding to any frame to be rendered in the video; Determine the complexity of each of the plurality of drawing calls corresponding to the picture to be rendered of any frame; Determining a rendering mode for each drawing call based on the predicted GPU utilization and the complexity of each drawing call, wherein the rendering mode includes any one of client-side rendering and server-side rendering; Based on the rendering mode of each drawing call, any frame of the to-be-rendered picture is rendered to determine the rendered picture; The determining the complexity of each of the multiple drawing calls corresponding to the picture to be rendered of any frame includes: Determine a rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture; Determining a complexity of each draw call based on a rendering type of each draw call and a graphics application program interface of each draw call; The determining, based on the predicted GPU utilization and the complexity of each drawing call, a rendering mode of each drawing call, comprises: Based on the predicted graphics processor utilization, the complexity of each drawing call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold, a rendering mode of each drawing call is determined; the preset first utilization threshold is greater than the preset second utilization threshold, and the preset first complexity threshold is greater than the preset second complexity threshold.

2. The method according to claim 1, characterized in that The determining of the rendering type of each of the multiple drawing calls corresponding to the picture to be rendered of any frame includes: By using a preset analysis tool, the video is analyzed offline to obtain rendering state characteristics of a draw call of any rendering type; If the rendering state characteristics of each draw call are the same as the rendering state characteristics of the draw call of any one rendering type, then determining the rendering type of each draw call is the any one rendering type; the rendering state characteristics of each draw call include a graphics application program interface characteristic of each draw call and a shader characteristic of each draw call.

3. The method according to claim 1, characterized in that The determining the complexity of each draw call based on the rendering type of each draw call and the graphics application program interface of each draw call includes: Determine, based on the rendering type of each draw call, an initial complexity of each draw call, wherein the initial complexity of each draw call is used to characterize a preliminary predicted rendering cost of each draw call; Determining, based on a graphics application program interface of each draw call, the number of models of each draw call, the model complexity of each draw call, and the number of shader instructions of each draw call; The complexity of each draw call is determined based on the initial complexity of each draw call, the number of models of each draw call, the model complexity of each draw call, and the number of shader instructions of each draw call, where the complexity of each draw call is used to characterize the predicted rendering cost of each draw call.

4. The method according to claim 3, characterized in that The determining, based on the graphics application program interface of each draw call, the number of models of each draw call, the model complexity of each draw call, and the number of shader instructions of each draw call comprises: Determining, through a graphics application programming interface of each draw call, the number of models of each draw call and the model complexity of each draw call; The number of shader instructions of each drawing call is determined through a reflection mechanism of the graphics application program interface.

5. The method according to claim 1, characterized in that The determining, based on the predicted GPU utilization, the complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold, a rendering mode of each draw call includes: If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each drawing call is greater than or equal to the first complexity threshold, determining that the rendering mode of each drawing call is the server-side rendering; If the predicted GPU utilization is less than the first utilization threshold, and the complexity of each drawing call is less than the first complexity threshold, determining the rendering mode of each drawing call to be the client rendering; If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is the server-side rendering; If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is less than the second complexity threshold, determining that the rendering mode of each draw call is the client rendering; If the predicted GPU utilization is less than the first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to the second utilization threshold, determining that the rendering mode of each draw call is the server-side rendering; If the predicted GPU utilization is less than the first utilization threshold, the complexity of each drawing call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than the second utilization threshold, determining that the rendering mode of each drawing call is the client rendering.

6. The method according to claim 1, characterized in that The rendering method based on each drawing call, rendering any frame of the to-be-rendered picture, and determining the rendered picture, includes: If the rendering mode of a part of the drawing calls among the drawing calls is the client rendering, and the rendering mode of another part of the drawing calls among the drawing calls is the server rendering, rendering the part of the drawing calls to obtain client rendering results, and receiving server rendering results for the other part of the drawing calls sent by the server; The client rendering result and the server rendering result are mixed and superimposed to determine a rendered image.

7. A rendering device, characterized in that: include: The first processing module is used to obtain the predicted graphics processor utilization corresponding to any frame to be rendered in the video; A second processing module is used to determine the complexity of each drawing call among a plurality of drawing calls corresponding to any frame of the to-be-rendered picture; A third processing module, configured to determine a rendering mode of each drawing call based on the predicted GPU utilization and the complexity of each drawing call, wherein the rendering mode includes any one of client-side rendering and server-side rendering; A fourth processing module, configured to render any frame of the to-be-rendered picture based on the rendering mode of each drawing call, and determine a rendered picture; The second processing module is specifically used for: Determine a rendering type of each of the multiple drawing calls corresponding to any frame of the to-be-rendered picture; Determining a complexity of each draw call based on a rendering type of each draw call and a graphics application program interface of each draw call; The third processing module is specifically used for: Determining a rendering mode for each draw call based on the predicted GPU utilization, the complexity of each draw call, a preset first utilization threshold, a preset second utilization threshold, a preset first complexity threshold, and a preset second complexity threshold; The preset first utilization threshold is greater than the preset second utilization threshold, and the preset first complexity threshold is greater than the preset second complexity threshold.

8. The rendering device according to claim 7, characterized in that: The second processing module is specifically used for: By using a preset analysis tool, the video is analyzed offline to obtain rendering state characteristics of a draw call of any rendering type; If the rendering state characteristics of each draw call are the same as the rendering state characteristics of the draw call of any one rendering type, then determining the rendering type of each draw call is the any one rendering type; the rendering state characteristics of each draw call include a graphics application program interface characteristic of each draw call and a shader characteristic of each draw call.

9. The rendering device according to claim 7, characterized in that: The second processing module is specifically used for: Determine, based on the rendering type of each draw call, an initial complexity of each draw call, wherein the initial complexity of each draw call is used to characterize a preliminary predicted rendering cost of each draw call; Determining, based on a graphics application program interface of each draw call, the number of models of each draw call, the model complexity of each draw call, and the number of shader instructions of each draw call; The complexity of each draw call is determined based on the initial complexity of each draw call, the number of models of each draw call, the model complexity of each draw call, and the number of shader instructions of each draw call, where the complexity of each draw call is used to characterize the predicted rendering cost of each draw call.

10. The rendering device according to claim 9, characterized in that: The second processing module is specifically used for: Determining, through a graphics application programming interface of each draw call, the number of models of each draw call and the model complexity of each draw call; The number of shader instructions of each drawing call is determined through a reflection mechanism of the graphics application program interface.

11. The rendering device according to claim 7, characterized in that: The third processing module is specifically used for: If the predicted GPU utilization is greater than or equal to the first utilization threshold, and the complexity of each drawing call is greater than or equal to the first complexity threshold, determining that the rendering mode of each drawing call is the server-side rendering; If the predicted GPU utilization is less than the first utilization threshold, and the complexity of each drawing call is less than the first complexity threshold, determining the rendering mode of each drawing call to be the client rendering; If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is greater than or equal to the second complexity threshold, determining that the rendering mode of each draw call is the server-side rendering; If the predicted GPU utilization is greater than or equal to the first utilization threshold, the complexity of each draw call is less than the first complexity threshold, and the complexity of each draw call is less than the second complexity threshold, determining that the rendering mode of each draw call is the client rendering; If the predicted GPU utilization is less than the first utilization threshold, the complexity of each draw call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is greater than or equal to the second utilization threshold, determining that the rendering mode of each draw call is the server-side rendering; If the predicted GPU utilization is less than the first utilization threshold, the complexity of each drawing call is greater than or equal to the first complexity threshold, and the predicted GPU utilization is less than the second utilization threshold, determining that the rendering mode of each drawing call is the client rendering.

12. The rendering device according to claim 7, characterized in that: The fourth processing module is specifically used for: If the rendering mode of a part of the drawing calls among the drawing calls is the client rendering, and the rendering mode of another part of the drawing calls among the drawing calls is the server rendering, rendering the part of the drawing calls to obtain client rendering results, and receiving server rendering results for the other part of the drawing calls sent by the server; The client rendering result and the server rendering result are mixed and superimposed to determine a rendered image.

13. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images