Image rendering method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses an image rendering method, apparatus, computer equipment and storage medium; obtains a pending rendering instruction corresponding to a pending frame; determines the predicted load state of a graphics processor according to the pending rendering instruction; when the predicted load state of the graphics processor is a high load state, updates the pending rendering instruction to a target rendering instruction; and renders the pending frame according to the target rendering instruction through the graphics processor to obtain a rendered image. This solution can adjust the load state of the graphics processor when rendering the pending frame, optimize the performance of the graphics processor, and also prevent the graphics processor from freezing when rendering the pending frame, thereby improving the smoothness of the image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image rendering method, apparatus, computer equipment and storage medium. Background Art

[0002] Graphics rendering technology is a hot topic in the field of intelligent terminals. Currently, most graphics images are rendered using graphics processors. In heavy-loaded scenarios such as large-scale 3D games, the performance of graphics processors is often insufficient.

[0003] Therefore, the problem of insufficient performance of the graphics processor is usually solved by upgrading the hardware, which is very costly. Therefore, how to optimize the performance of the graphics processor is a major problem currently faced. Summary of the invention

[0004] The embodiments of the present application provide an image rendering method, apparatus, computer equipment and storage medium, which can adjust the load state of a graphics processor when rendering a frame to be processed and optimize the performance of the graphics processor.

[0005] An embodiment of the present application provides an image rendering method, including: obtaining a pending rendering instruction corresponding to a pending frame; determining a predicted load state of a graphics processor according to the pending rendering instruction; when the predicted load state of the graphics processor is a high load state, updating the pending rendering instruction to a target rendering instruction; and rendering the pending frame according to the target rendering instruction by the graphics processor to obtain a rendered image.

[0006] An embodiment of the present application also provides an image rendering device, including: an acquisition unit, used to acquire a pending rendering instruction corresponding to a pending frame; a determination unit, used to determine a predicted load state of a graphics processor based on the pending rendering instruction; an update unit, used to update the pending rendering instruction to a target rendering instruction when the predicted load state of the graphics processor is a high load state; and a rendering unit, used to render the pending frame according to the target rendering instruction through the graphics processor to obtain a rendered image.

[0007] An embodiment of the present application also provides a computer device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the image rendering methods provided in the embodiments of the present application.

[0008] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in any one of the image rendering methods provided in the embodiment of the present application.

[0009] The embodiment of the present application can obtain the pending rendering instruction corresponding to the pending frame; determine the predicted load state of the graphics processor according to the pending rendering instruction; when the predicted load state of the graphics processor is a high load state, update the pending rendering instruction to a target rendering instruction; and render the pending frame according to the target rendering instruction through the graphics processor to obtain a rendered image. In the present application, by predicting the load state of the graphics processor processing the pending frame, updating the pending rendering instruction in a high load state, and using the target rendering instruction for rendering, the load state of the graphics processor when rendering the pending frame is adjusted, the performance of the graphics processor is optimized, and the graphics processor does not freeze when rendering the pending frame, thereby improving the smoothness of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 is a scene schematic diagram of the image rendering method provided in an embodiment of the present application;

[0012] Figure 2 It is a flowchart of an image rendering method provided in an embodiment of the present application;

[0013] Figure 3 It is a schematic diagram of a rendering process applied to a terminal provided in an embodiment of the present application;

[0014] Figure 4 is a flowchart of another image rendering method provided in an embodiment of the present application;

[0015] Figure 5 is a structural schematic diagram of an image rendering device provided in an embodiment of the present application;

[0016] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0018] Embodiments of the present application provide an image rendering method, apparatus, computer device, and storage medium.

[0019] The image rendering device can be integrated into an electronic device, which can be a terminal, a server, or the like. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer (PC), etc. The server can be a single server or a server cluster consisting of multiple servers.

[0020] In some embodiments, the image rendering device may also be integrated into multiple electronic devices. For example, the image rendering device may be integrated into multiple servers, and the image rendering method of the present application may be implemented by multiple servers.

[0021] In some embodiments, the server may also be implemented in the form of a terminal.

[0022] For example, refer to Figure 1 In some embodiments, a scene diagram of an image rendering system is provided, and the image rendering system may include a terminal 1000, a server 2000, and a network 3000. The terminal is integrated with an image rendering device, the server stores image data to be rendered, and the image data to be rendered includes frames to be processed. The server and the terminal can exchange data through the network.

[0023] Among them, the terminal can obtain image data to be rendered from the server through the network, and obtain the frame to be processed from the image data to be rendered. After obtaining the frame to be processed, obtain the rendering instruction to be processed corresponding to the frame to be processed; determine the predicted load state of the graphics processor according to the rendering instruction to be processed; when the predicted load state of the graphics processor is a high load state, update the rendering instruction to be processed to the target rendering instruction; render the frame to be processed according to the target rendering instruction through the graphics processor, and obtain the rendered image for sending the frame to be processed to the terminal.

[0024] Among them, the network is used for data transmission between the server and the target device. The network can be a wireless network or a wired network. For example, the wireless network is a wireless local area network (WLAN), a local area network (LAN), a cellular network, a 2G network, a 3G network, a 4G network, a 5G network, etc.

[0025] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0026] In this embodiment, an image rendering method is provided, such as Figure 2 As shown, the specific process of the image rendering method can be as follows:

[0027] 110. Obtain the pending rendering instruction corresponding to the pending frame.

[0028] The to-be-processed frame refers to a frame to be rendered by a graphics processor, and may be one or more frames to be currently rendered by the graphics processor.

[0029] In some implementations, in order to reduce the amount of data processed at a time, simplify the rendering instruction processing process, reduce the resource overhead of the graphics processor, and avoid freezes and data errors, the frame to be processed is the current frame. The current frame refers to a frame that is currently being rendered by the graphics processor.

[0030] Rendering refers to the process of generating an image from a model during computer drawing, which is generally performed by a graphics processing unit (GPU), which is a microprocessor specifically used for image computing and is often used for graphics rendering. For example, in this embodiment, a central processing unit (CPU) may be deployed in the terminal. When the terminal runs a game application, the game program instructs the CPU to encapsulate game resources in rendering instructions and send rendering instructions to the graphics processing unit, so that the graphics processing unit processes these rendering instructions and finally generates the game screen. For example, Figure 3 As shown, on the one hand, in the software drawing process, the central processing unit performs software drawing through Measure, Layout, Draw and Skia. On the other hand, in the hardware drawing process, the central processing unit generates rendering commands through Measure, Layout and Draw. The central processing unit submits the generated rendering commands to the graphics processor through the OpenGL ES (OpenGL for Embedded Systems) module. Each rendering command corresponds to the OpenGL ES API. The graphics processor renders graphics according to the rendering instructions, and displays the pictures on the screen after rasterizing the software and hardware drawing results. The rendering process in the embodiment of the present application is Figure 3 The hardware drawing process in .

[0031] The pending rendering instructions are instructions for instructing the graphics processor to render the screen according to the pending frames. The pending rendering instructions reflect the workload of the GPU in processing the pending frames. When the CPU generates rendering instructions based on program resources, the rendering instructions are collected in units of frames to be rendered by the GPU. In some embodiments, before step 110, the step of generating pending rendering instructions corresponding to the pending frames may also be included.

[0032] 120. Determine a predicted load state of a graphics processor according to the rendering instructions to be processed.

[0033] The load state refers to the GPU load working state, and the load state may include but is not limited to at least one of a high load state, a low load state, a normal load state, and the like. For example, a high load state can be used to characterize a state where the GPU load is higher than a first load threshold, a normal load state can be used to characterize a state where the GPU load is between a first load threshold and a second load threshold, and a low load state can be used to characterize a state where the GPU load is lower than a second load threshold, wherein the first load threshold is greater than the second load threshold, and the first load threshold and the second load threshold can be set according to application scenarios or experience. In actual applications, the GPU load can be represented by characteristic values ​​related to the GPU load, such as GPU occupancy, GPU operating frequency, or the number of APIs, and correspondingly, the first load threshold and the second load threshold are also preset GPU occupancy or the number of APIs, and the like.

[0034] The predicted load state refers to the load state of the graphics processor when processing the rendering instructions corresponding to the frame to be processed, which is predicted based on the current or historical load information of the GPU, or based on the parameters of the GPU, or a combination of the two, and so on.

[0035] In some embodiments, the predicted load state may include a high load state and a low load state. When the graphics processor is in a high load state, it means that the graphics processor is overloaded and prone to abnormal situations such as freezing or frame drops. When the graphics processor is in a low load state, it means that the graphics processor load is normal and will not experience abnormal situations such as freezing or frame drops. For example, the high load state can be used to characterize a state where the GPU load is higher than a first load threshold, and the low load state can be used to characterize a state where the GPU load is lower than the first load threshold.

[0036] In some implementations, in order to facilitate determining the predicted load state of the graphics processor and improve processing efficiency, step 120 may include steps 1.1 to 1.2, as follows:

[0037] 1.1. Obtain the first parameter corresponding to the rendering instruction to be processed;

[0038] 1.2. Determine the predicted load state of the graphics processor according to the first parameter and the threshold condition.

[0039] Among them, the first parameter refers to the parameter when the API (Application Programming Interface) corresponding to the rendering instruction is called, which can be obtained by HOOKing the API of the frame to be processed. The first parameter can include the function name and other parameters and the like.

[0040] The threshold condition refers to a condition used to determine the load state of the graphics processor. For example, the load state of the graphics processor can be determined based on the type, quantity, or type and quantity of the first parameter, and can also be preset according to needs or experience. For example, since there are generally multiple first parameters corresponding to an image frame, the first parameters may include type one and type two. All first parameters are classified according to these two types, and it is obtained that there are m first parameters of type one and n first parameters of type two. The threshold condition may be that when there are more than a first parameters of type one, the load state of the graphics processor is determined to be a high load state, or when there are first parameters of type two, the load state of the graphics processor is determined to be a high load state, and so on.

[0041] By setting the threshold condition, the first parameter corresponding to the frame to be processed can be compared with the threshold condition to determine whether the workload of the graphics processor when executing the rendering instruction of the frame to be processed exceeds the workload determined by the threshold condition.

[0042] In some embodiments, the number of the first parameter used can reflect the workload of the API interface call when executing the rendering instruction, so as to more intuitively use the first parameter and the threshold condition to characterize the workload of the graphics processor. The threshold condition may include when the number of the first parameter is greater than a preset value, determining the predicted load state of the graphics processor as a high load state, and when the number of the first parameter is less than or equal to the preset value, determining the predicted load state of the graphics processor as a low load state. Step 1.2 may include the step of: determining the predicted load state of the graphics processor according to the number of the first parameter and the threshold condition.

[0043] In some implementations, steps 2.1 to 2.4 may be further included before step 120, as follows:

[0044] 2.1. Get the preset frame;

[0045] 2.2. Generate preset rendering instructions corresponding to the preset frame;

[0046] 2.3. Get the second parameter corresponding to the preset rendering instruction;

[0047] 2.4. Determine the threshold condition based on the second parameter.

[0048] The preset frame refers to a frame used to determine the threshold condition. The preset frame can be preset according to actual needs or experience. For example, the preset frame is determined according to the data volume of the frame, and the frame with the data volume equal to the preset data volume is used as the preset frame. The preset data volume can be determined according to the working frequency of the graphics processor, or according to the preset application scenario and the frame rate threshold corresponding to the scenario, or can be preset, etc. The preset frame can include one frame or multiple frames. In some embodiments, the number of frames of the preset frame is the same as the number of frames to be processed. For example, when the frame to be processed is one frame, the preset frame is also one frame.

[0049] The preset rendering instruction refers to an instruction used to instruct the graphics processor to render the screen according to the preset frame. The second parameter refers to the parameter when the API (Application Programming Interface) corresponding to the preset rendering instruction is called, which can be obtained by HOOKing the API of the preset frame. The second parameter may include a function name and other parameters. The method for determining the preset rendering instruction and the second parameter is the same as the method for determining the instruction to be processed and the first parameter.

[0050] Before rendering the frame to be processed, the threshold condition is determined by the preset frame to judge the frame to be processed and determine the load state of the graphics processor when processing the frame to be processed. Different preset frames can be selected according to the application scenario to achieve flexible judgment of the load state of the graphics processor, and the obtained threshold condition is highly correlated with the frame, thereby improving the matching degree between the threshold condition and the characteristics of the frame to be processed, making the judgment result of the load state of the graphics processor more reasonable.

[0051] In step 2.4, the second parameter can be used as a threshold reference to determine the threshold condition, such as using the number of second parameters as the preset value in the threshold condition described above, or multiple second parameters corresponding to multiple preset frames can be obtained, and the threshold reference can be obtained based on the multiple second parameters to determine the threshold condition, for example, using the average value of the number of multiple second parameters as the preset value in the threshold condition described above.

[0052] In some implementations, in order to simplify the processing procedure and improve processing efficiency, the preset value in the threshold condition may be the number of the second parameter.

[0053] In some implementations, the preset frame is a frame that meets the preset operating conditions. By acquiring the candidate frames and determining the preset frame from the candidate frames, the rationality of the process of determining the candidate frames can be increased, making the result of the load state judgment of the graphics processor more reasonable. Step 2.1 can include steps 2.1.1 to 2.1.3, as follows:

[0054] 2.1.1. Obtain multiple candidate frames;

[0055] 2.1.2. Render each candidate frame separately to obtain the graphics processor operation information for each candidate frame;

[0056] 2.1.3. Determine a preset frame from multiple candidate frames according to the graphics processor operation information of all candidate frames and preset operation conditions.

[0057] Among them, the candidate frame refers to the frame used to determine the preset frame. The candidate frame can be pre-set according to actual needs or experience, or randomly selected. For example, a plurality of frames with the same scene as the frame to be processed are randomly selected as candidate frames. If the frame to be processed is a frame of a 3D game screen, a plurality of frames can be randomly selected from the 3D game screen as candidate frames.

[0058] The graphics processor operation information refers to information when the graphics processor processes each candidate frame, and may include at least one of frequency, freeze condition, occupancy rate, processing time, and the like.

[0059] The preset operating conditions refer to pre-set conditions used to determine the operating information of the graphics processor, which may include conditions for determining whether the frequency of the graphics processor when rendering candidate frames is higher than the preset frequency, whether the graphics processor is stuck when rendering candidate frames, or whether the occupancy rate of the graphics processor when rendering candidate frames is higher than the preset occupancy rate, or whether the processing time of the graphics processor when rendering candidate frames is higher than the preset time, etc.

[0060] In some embodiments, in order to facilitate the determination of the predicted load state of the graphics processor and improve processing efficiency, the graphics processor operation information may include a jamming situation and operation parameters, and the operation parameters may include at least one of frequency, occupancy rate, processing time, etc. The preset operation conditions may include determining a candidate frame that is jammed during rendering and meets the preset parameter conditions as a preset frame. Step 2.1.3 may include the steps of: determining the candidate frame that is jammed during rendering as a high-load frame based on the graphics processor operation information of all candidate frames; determining the preset frame from all high-load frames based on the operation parameters and preset parameter conditions corresponding to the high-load frame, for example, a high-load frame whose operation parameter value is the minimum among the parameter values ​​corresponding to all high-load frames is determined as the preset frame, such as determining a high-load frame with the shortest processing time as the preset frame, or performing weight calculation on at least two of the processing time, frequency, and occupancy rate, and determining the high-load frame corresponding to the calculated minimum value as the preset frame.

[0061] 130. When the predicted load state of the graphics processor is a high load state, update the pending rendering instruction to a target rendering instruction.

[0062] The target rendering instruction refers to an instruction obtained by updating the pending rendering instruction and used by the graphics processor to render the picture according to the pending frame. The method for updating the pending instruction can be to update the pending rendering instruction to the target rendering instruction by reducing the rendering resolution. By updating the pending rendering instruction and using the target rendering instruction for rendering, the load state of the graphics processor when rendering the pending frame is adjusted, and the performance of the graphics processor is optimized.

[0063] In some implementations, in order to reduce the amount of data to be processed by the image processor and improve processing efficiency while maintaining the quality of the rendered image, the rendering instruction to be processed includes a plurality of sub-instructions to be processed, and step 130 may include steps 3.1 to 3.4, as follows:

[0064] 3.1. Determine a first target object and a second target object in a frame to be processed;

[0065] 3.2. Cut the sub-instructions to be processed corresponding to the first target object to obtain the sub-instructions to be processed for the first target;

[0066] 3.3. Merge the pending sub-instructions corresponding to the second target object to obtain the second target pending sub-instructions;

[0067] 3.4. According to the first target pending sub-instruction and the second target pending sub-instruction, the pending rendering instruction is updated to the target rendering instruction.

[0068] The first target object refers to a non-key object in the image of the frame to be processed. There are key objects in the image of the frame to be processed that are used to represent the image scene. The key objects can be pre-set according to the scene. Objects other than the key objects in the image of the frame to be processed are non-key objects, and the non-key objects do not affect the expression effect of the image of the frame to be processed. For example, when the frame to be processed is a frame of a 3D game screen, the player characters in the screen and the buildings in the near view are key objects, and the buildings and plants in the distant view of the screen are non-key objects.

[0069] Among them, the second target object refers to the same object in the picture of the frame to be processed. For example, in the picture of the frame to be processed, there are multiple static models with the same material that need to be rendered. The model is in a fixed position during the application operation and will not be scaled. The static model is the second object. The rendering instructions to be processed include sub-instructions to be processed corresponding to the first target object and the second target object, and also include sub-instructions to be processed corresponding to key objects. It should be noted that there can be multiple second target objects in the picture of the frame to be processed, and each second target object can include multiple identical second target sub-objects, and each second target object is different. For example, there are multiple identical houses and multiple identical trees in the 3D game picture. The multiple identical houses are one second target sub-object, and the multiple identical trees are another second target sub-object.

[0070] Among them, instruction clipping refers to the operation of reducing the amount of data to be processed when rendering the first target object. For example, it can be a method such as reducing the number of vertices, deleting the pending sub-instructions corresponding to non-critical objects, or treating them as empty instructions. For example, by reducing the number of vertices to process the second target object, a new rendering instruction corresponding to the second target object is obtained, and the second target pending sub-instruction of the new rendering instruction is deleted. When the pending sub-instructions corresponding to the non-critical objects are deleted or treated as empty instructions, the second target pending sub-instruction is an empty instruction.

[0071] The instruction merging refers to merging the same instructions corresponding to the second target object. For example, merging the same instructions for multiple identical second target sub-objects, or merging the same instructions for one second target sub-object to obtain the second target to-be-processed sub-instruction, etc. The second target to-be-processed sub-instruction may be a batch processing instruction for the second target object.

[0072] The first target object is determined by non-critical objects in the picture of the frame to be processed, and the second target object is determined by the same object, and the pending sub-instructions corresponding to the first target object and the second target object are processed respectively to obtain the first target pending sub-instruction and the second target pending sub-instruction. The pending sub-instruction corresponding to the first target object in the pending rendering instruction can be replaced with the first target pending sub-instruction, and the pending sub-instruction corresponding to the second target object in the pending rendering instruction can be replaced with the second target pending sub-instruction to obtain the target rendering instruction.

[0073] In some implementations, by simplifying the vertex data of the first target object, the first target to-be-processed sub-instruction is obtained, the amount of data to be processed by the image processor is reduced, and the processing efficiency is improved. Step 3.2 may include steps 3.2.1 to 3.2.3, as follows:

[0074] 3.2.1. Determine the initial vertex data corresponding to the first target object;

[0075] 3.2.2, determine the target vertex data from the initial vertex data through a clustering algorithm;

[0076] 3.2.3. Update the to-be-processed sub-instruction corresponding to the first target object according to the target vertex data to obtain the first target to-be-processed sub-instruction.

[0077] The initial vertex data refers to the position information of all vertices of the first target object in the frame to be processed, and the initial vertex data includes the position information of multiple vertices. The graphics processor sends the initial vertex data to the graphics processor, and the image of the first object can be obtained by rendering the initial vertex data.

[0078] The clustering algorithm may be a K-Means clustering algorithm or a Rossignac vertex clustering algorithm, etc. For example, the Rossignac vertex clustering algorithm is used to detect and merge clusters of adjacent vertices to simplify the mesh, and each cluster is replaced by a representative vertex, which may be the center of the vertex cluster or the vertex with the maximum weight in the cluster, and all representative vertices constitute the target vertex data.

[0079] By acquiring vertex data of the first target object, obtaining target vertex data composed of representative vertices by a clustering algorithm, and generating a first target to-be-processed sub-instruction according to the target vertex data, the to-be-processed sub-instruction corresponding to the first target object is updated to the first target to-be-processed sub-instruction.

[0080] In some implementations, in order to simplify the rendering process of multiple identical objects, reduce the amount of data to be processed by the image processor, and improve processing efficiency, the second target object includes multiple identical second target sub-objects, and step 3.3 may include steps 3.3.1 to 3.3.3, as follows:

[0081] 3.3.1. Obtain the pending sub-instructions corresponding to any second target sub-object;

[0082] 3.3.2. Generate multi-instance rendering instructions corresponding to multiple second target sub-objects according to the sub-instruction to be processed and multiple identical second target sub-objects;

[0083] 3.3.3. Determine the multi-instance rendering instruction as the second target sub-instruction to be processed.

[0084] The multi-instance rendering instruction refers to an instruction used to execute multiple identical rendering instructions, and the multiple identical rendering instructions correspond one-to-one to multiple identical second target sub-objects.

[0085] When there are multiple identical objects to be rendered in the frame to be processed, during the original rendering, data values ​​need to be transmitted to the graphics processor multiple times to render each object separately. However, the use of multi-instance rendering instructions can transmit as much vertex data as possible in one drawing instruction, reducing the call of drawing instructions, that is, multiple objects can be drawn by transmitting data once.

[0086] It should be noted that steps 3.3.1 to 3.3.2 are processes executed for one second target object. When there are multiple second target objects, steps 3.3.1 to 3.3.2 are executed for the multiple second target objects respectively.

[0087] In some implementations, in order to adjust the working state of the graphics processor and reduce the load of the graphics processor, after step 130, the process may further include the step of adjusting the working frequency of the graphics processor to a target working frequency.

[0088] The target operating frequency refers to the preset operating frequency of the graphics processor, which can be preset according to actual needs or experience. The target operating frequency can be higher than the frequency of the graphics processor at the current moment, or can be determined according to the frequency of the graphics processor when processing a preset frame, such as using the frequency as the target operating frequency, or using it as the target operating frequency after weight calculation, etc. In some embodiments, in order to maximize the use of resources of the graphics processor, the target operating frequency is the highest frequency of the graphics processor.

[0089] 140. Render the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

[0090] The rendered image refers to an image containing a picture of the frame to be processed.

[0091] After obtaining the target rendering instruction, the graphics processor processes the target rendering instruction to generate a picture of the frame to be processed. It should be noted that the embodiment of the present application can be used to render an image with multiple consecutive frames. When obtaining the rendering instruction corresponding to each frame, the target rendering instruction can be obtained by the method of the embodiment of the present application. After processing the target rendering instruction corresponding to each frame of the image with multiple consecutive frames, the picture of the image is obtained.

[0092] The image rendering solution provided by the embodiment of the present application can be applied in various image rendering scenarios. For example, taking game rendering as an example, the pending rendering instructions corresponding to the pending frame can be obtained; according to the pending rendering instructions, the predicted load state of the graphics processor is determined; when the predicted load state of the graphics processor is a high load state, the pending rendering instructions are updated to the target rendering instructions; and the pending frame is rendered by the graphics processor according to the target rendering instructions to obtain a rendered image.

[0093] The method provided in the embodiment of the present application can predict the load state of the graphics processor when processing the frames to be processed, update the pending rendering instructions when in a high load state, and use the target rendering instructions for rendering to adjust the load state of the graphics processor when rendering the frames to be processed, thereby optimizing the performance of the graphics processor, and preventing the graphics processor from being stuck when rendering the frames to be processed, thereby improving the smoothness of the image.

[0094] The method described in the above embodiment will be further described in detail below.

[0095] In this embodiment, the method of the embodiment of the present application is described in detail by taking the application of the image rendering of the present application to the TV screen rendering as an example.

[0096] like Figure 4 As shown, the specific process of an image rendering method is as follows:

[0097] 210. Obtain a pending rendering instruction corresponding to a pending frame, where the pending rendering instruction includes a plurality of pending sub-instructions.

[0098] The pending frame is a frame, which can be a frame to be rendered currently. First, the rendering command (HOOK API) of the pending frame is obtained, that is, each frame is marked before and after rendering, and the function name and parameters of each frame rendering command are obtained through HOOK each frame API. When obtaining the pending rendering instructions of the pending frame, the pending rendering instructions can be counted to provide data support for subsequent optimization, such as counting the number or type of APIs corresponding to the pending sub-instructions in the pending rendering instructions, etc., so that the statistical results can be judged according to the threshold conditions to determine the load status of the pending rendering instructions.

[0099] It should be noted that the specific implementation methods of the terms and some steps in this embodiment can be found in the corresponding parts of the aforementioned embodiments and will not be repeated here.

[0100] 220. Determine the predicted load state of the graphics processor according to the rendering instruction to be processed. When the predicted load state of the graphics processor is a high load state, execute steps 230 to 270. When the predicted load state of the graphics processor is a low load state, execute step 280.

[0101] The predicted load state of the graphics processor can be determined by:

[0102] 1. Divide the GPU load level: First, use the frame rate detection tool to detect the frame rate of each scene, and pre-set the GPU load level for the application scenario (including the frame rate threshold for each application scenario), where a normal load indicates that the system has no stuck frames, and an excessive load indicates that the system has stuck frames. For example, when the corresponding terminal is a TV, when the frame rate of the graphics processor rendering the TV screen is greater than or equal to the frame rate threshold, it can be determined as a normal load, and when it is less than the preset frame rate, it is an excessive load, and the frame corresponding to the rendered screen is determined as a candidate frame. At this time, the frame rate of the graphics processor rendering the TV screen is the GPU load used to distinguish between a high load state and a low load state in the aforementioned embodiment, and the frame rate threshold is the first load threshold.

[0103] 2. The corresponding threshold value T is obtained by using the divided load levels. The threshold value is determined by counting the number of candidate frame rendering command APIs corresponding to normal load scenes and overload scenes respectively. For example, the maximum number of APIs in the normal load scene is used as the threshold, and the corresponding frame is the preset frame, or the minimum number of APIs in the overload scene is used as the threshold, and the corresponding frame is the preset frame.

[0104] 3. Determine whether the rendering load of the frame to be processed is too heavy according to the threshold value T. If the load is too heavy, it is a high load state, otherwise it is a low load state. In this way, the predicted load state of the frame to be processed that is greater than the threshold value T is determined as a high load state, and the predicted load state of the frame to be processed that is less than or equal to the threshold value T is determined as a low load state, thereby realizing the prediction of the load state of the graphics processor when processing the frame to be processed. That is, it is predicted that when processing the frame to be processed that is greater than the threshold value T, the frame rate of the graphics processor is less than the preset frame rate (that is, it is a high load state), and when processing the frame to be processed that is less than the threshold value T, the frame rate of the graphics processor is greater than or equal to the preset frame rate (that is, it is a low load state).

[0105] By judging the predicted load status of the graphics processor, if the load of the frame to be processed is normal, no processing is required. If the load is too heavy, frequency modulation, instruction clipping, instruction merging, etc. are performed.

[0106] 230. When the predicted load state of the graphics processor is a high load state, adjust the operating frequency of the graphics processor to a maximum frequency.

[0107] When the GPU's predicted load is high, the pending frame will freeze and the GPU frequency needs to be adjusted. Since the TV does not need to consider power consumption, the GPU frequency is adjusted to the highest level until the pending frame is rendered. The rendering of a frame starts with DequeueBuffer and ends with the QueueBuffer method.

[0108] 240. When the predicted load state of the graphics processor is a high load state, perform instruction pruning on the to-be-processed sub-instructions corresponding to the first target object to obtain the first target to-be-processed sub-instructions.

[0109] Instruction clipping can be performed for small objects with a long viewing distance in 3D scenes. Without reducing the performance effect, the number of triangles, that is, the number of vertices, can be minimized to improve the efficiency of graphics drawing. According to the rule of 3D imaging that large objects appear closer and smaller objects appear farther away, the minimum Z coordinate of the object in the scene is taken to determine whether the object belongs to the distant view in the scene. The method of reducing vertices adopts a merging mechanism. For example, the vertex clustering algorithm such as Rossignac is used to detect and merge clusters of adjacent vertices to simplify the mesh. Each cluster is replaced by a representative vertex. This representative vertex may be the center of the vertex cluster or the vertex with the largest weight in the cluster (the vertex weight is defined to emphasize the relative visual importance). Finally, the overlapping or degenerate edges or triangles caused by the clustering operation are removed to obtain the first target sub-instruction to be processed.

[0110] In some implementations, the instruction clipping method may also be replaced by selecting an empty instruction or reducing the number of vertices.

[0111] 250. When the predicted load state of the graphics processor is a high load state, merge the to-be-processed sub-instructions corresponding to the second target object to obtain second target to-be-processed sub-instructions.

[0112] Instruction merging is achieved by batch processing static objects in the scene. For example, multiple static models of the same material need to be drawn uniformly in the scene. The model is in a fixed position during the application running process and will not be scaled. In addition, the original rendering is done using the glElements API. You can replace glElements with glDrawElementsInstance for rendering. Therefore, you only need to pass one instance to the graphics processor and set the rendering quantity parameters to render multiple models.

[0113] Batch processing can use glDrawElementsInstance instead of glDrawElements to render static objects in the scene, and adjust the graphics processor frequency to the highest before rendering, and the duration is from before rendering to the end of rendering.

[0114] 260. Update the pending rendering instruction to a target rendering instruction according to the first target pending sub-instruction and the second target pending sub-instruction.

[0115] The to-be-processed sub-instruction corresponding to the first target object in the to-be-processed rendering instruction is replaced with the first target to-be-processed sub-instruction, and the to-be-processed sub-instruction corresponding to the second target object in the to-be-processed rendering instruction is replaced with the second target to-be-processed sub-instruction to obtain the target rendering instruction.

[0116] 270. After adjusting the graphics processor to the highest frequency, the frame to be processed is rendered according to the target rendering instruction to obtain a rendered image.

[0117] 280. When the predicted load state of the graphics processor is a low load state, the graphics processor renders the frame to be processed according to the rendering instruction to be processed to obtain a rendered image.

[0118] As can be seen from the above, the embodiment of the present application predicts the load state of the graphics processor and performs jam detection in frames. When the graphics processor predicts that the load state is a high load state, after initiating a frequency increase request, the graphics processor's operating frequency is adjusted to the highest frequency in real time to schedule resources in real time. According to the obtained rendering commands, rendering instruction clipping and batch processing are performed to reduce the workload and load of the graphics processor, optimize the performance of the graphics processor, and improve the smoothness of the image.

[0119] In order to better implement the above method, the embodiment of the present application also provides an image rendering device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0120] For example, in this embodiment, the method of the embodiment of the present application is described in detail by taking the image rendering device being specifically integrated in a terminal as an example.

[0121] For example, Figure 5 As shown, the image rendering device may include an acquisition unit 310, a determination unit 320, an update unit 330, and a rendering unit 340, as follows:

[0122] (I) Acquisition Unit 310

[0123] Used to obtain pending rendering instructions corresponding to the pending frame.

[0124] (ii) Determination unit 320

[0125] Used to determine the predicted load state of the graphics processor based on pending rendering instructions.

[0126] In some implementations, the rendering instruction to be processed includes a plurality of sub-instructions to be processed, and the determination unit 320 may be specifically used in steps 4.1 to 4.2 as follows:

[0127] 4.1. Obtain the first parameter corresponding to the rendering instruction to be processed;

[0128] 4.2. Determine the predicted load state of the graphics processor according to the first parameter and the threshold condition.

[0129] In some implementations, the determination unit 320 may also be used in steps 5.1 to 5.4 as follows:

[0130] 5.1. Get the preset frame;

[0131] 5.2. Generate preset rendering instructions corresponding to the preset frame;

[0132] 5.3. Get the second parameter corresponding to the preset rendering instruction;

[0133] 5.4. Determine the threshold condition based on the second parameter.

[0134] (III) Update Unit 330

[0135] When the predicted load state of the graphics processor is a high load state, the pending rendering instruction is updated to a target rendering instruction.

[0136] In some implementations, the rendering instruction to be processed includes a plurality of sub-instructions to be processed, and the updating unit 330 may be specifically used in steps 6.1 to 6.4 as follows:

[0137] 6.1. Determine a first target object and a second target object in a frame to be processed;

[0138] 6.2. Cut the to-be-processed sub-instructions corresponding to the first target object to obtain the to-be-processed sub-instructions of the first target;

[0139] 6.3. Merge the pending sub-instructions corresponding to the second target object to obtain the second target pending sub-instructions;

[0140] 6.4. According to the first target pending sub-instruction and the second target pending sub-instruction, the pending rendering instruction is updated to the target rendering instruction.

[0141] In some embodiments, step 6.2 may include steps 6.2.1 to 6.2.3, as follows:

[0142] 6.2.1. Determine the initial vertex data corresponding to the first target object;

[0143] 6.2.2. Determine the target vertex data from the initial vertex data through a clustering algorithm;

[0144] 6.2.3. Update the pending sub-instruction corresponding to the first target object according to the target vertex data to obtain the first target pending sub-instruction.

[0145] In some embodiments, the second target object includes a plurality of identical second target sub-objects, and step 6.3 may include steps 6.3.1 to 6.3.2, as follows:

[0146] 6.3.1. Obtain the pending sub-instructions corresponding to any second target sub-object;

[0147] 6.3.2. Generate multi-instance rendering instructions corresponding to multiple second target sub-objects according to the sub-instruction to be processed and multiple identical second target sub-objects;

[0148] 6.3.3. Determine the multi-instance rendering instruction as the second target sub-instruction to be processed.

[0149] In some implementations, after step 130, the method may further include the step of adjusting the operating frequency of the graphics processor to a target operating frequency.

[0150] (IV) Rendering unit 340

[0151] It is used to render the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

[0152] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0153] Therefore, the embodiment of the present application can adjust the load state of the graphics processor when rendering the frames to be processed, optimize the performance of the graphics processor, and prevent the graphics processor from being stuck when rendering the frames to be processed, thereby improving the smoothness of the image.

[0154] Correspondingly, an embodiment of the present application also provides a computer device, which may be a terminal or a server, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer, a personal digital assistant (PDA), or other terminal devices.

[0155] like Figure 6 As shown, Figure 6The schematic diagram of the structure of the computer device provided in the embodiment of the present application is that the computer device 400 includes a processor 410 having one or more processing cores, a memory 420 having one or more computer-readable storage media, and a computer program stored in the memory 420 and executable on the processor. The processor 410 is electrically connected to the memory 420. It can be understood by those skilled in the art that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0156] The processor 410 is the control center of the computer device 400. It uses various interfaces and lines to connect various parts of the entire computer device 400, executes various functions of the computer device 400 and processes data by running or loading software programs and / or modules stored in the memory 420, and calling data stored in the memory 420, thereby monitoring the computer device 400 as a whole.

[0157] In the embodiment of the present application, the processor 410 in the computer device 400 will load instructions corresponding to the processes of one or more application programs into the memory 420 according to the following steps, and the processor 410 will run the application programs stored in the memory 420 to implement various functions:

[0158] Obtain a pending rendering instruction corresponding to the pending frame; determine a predicted load state of the graphics processor according to the pending rendering instruction; when the predicted load state of the graphics processor is a high load state, update the pending rendering instruction to a target rendering instruction; render the pending frame according to the target rendering instruction through the graphics processor to obtain a rendered image.

[0159] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0160] Optional, such as Figure 6 As shown, the computer device 400 further includes: a touch screen 430, a radio frequency circuit 440, an audio circuit 450, an input unit 460, and a power supply 470. The processor 410 is electrically connected to the touch screen 430, the radio frequency circuit 440, the audio circuit 450, the input unit 460, and the power supply 470, respectively. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0161] The touch display screen 430 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 430 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the computer device, which may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or accessory such as a finger, stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 410, and can receive the command sent by the processor 410 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 410 to determine the type of touch event, and then the processor 410 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 430 to realize the input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 430 can also be used as a part of the input unit 460 to realize the input function.

[0162] In the embodiment of the present application, the processor 410 executes the game application to generate a graphical user interface on the touch screen 430. The virtual scene on the graphical user interface includes at least one skill control area, and the skill control area includes at least one skill control. The touch screen 430 is used to present the graphical user interface and receive operation instructions generated by the user acting on the graphical user interface.

[0163] The radio frequency circuit 440 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other computer devices through wireless communication, and to send and receive signals between the network device or other computer devices.

[0164] The audio circuit 450 can be used to provide an audio interface between the user and the computer device through a speaker and a microphone. The audio circuit 450 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 450 and converted into audio data, and then the audio data is output to the processor 410 for processing, and then sent to another computer device through the radio frequency circuit 440, or the audio data is output to the memory 420 for further processing. The audio circuit 450 may also include an earphone jack to provide communication between an external headset and the computer device.

[0165] The input unit 460 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0166] The power supply 470 is used to supply power to various components of the computer device 400. Optionally, the power supply 470 can be logically connected to the processor 410 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 470 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0167] although Figure 6 Not shown, the computer device 400 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] From the above, it can be seen that the computer device provided in this embodiment can adjust the load state of the graphics processor when rendering the frames to be processed, optimize the performance of the graphics processor, and prevent the graphics processor from being stuck when rendering the frames to be processed, thereby improving the smoothness of the image.

[0170] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0171] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any image rendering method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0172] Obtain a pending rendering instruction corresponding to the pending frame; determine a predicted load state of the graphics processor according to the pending rendering instruction; when the predicted load state of the graphics processor is a high load state, update the pending rendering instruction to a target rendering instruction; render the pending frame according to the target rendering instruction through the graphics processor to obtain a rendered image.

[0173] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0174] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] Since the computer program stored in the storage medium can execute the steps in any image rendering method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image rendering method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0176] The above is a detailed introduction to an image rendering method, device, storage medium and computer device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image rendering method, characterized in that: include: Get the pending rendering instructions corresponding to the pending frame; Determining a predicted load state of a graphics processor according to the rendering instruction to be processed; When the predicted load state of the graphics processor is a high load state, updating the to-be-processed rendering instruction to a target rendering instruction; Rendering the frame to be processed according to the target rendering instruction by the graphics processor to obtain a rendered image; The to-be-processed rendering instruction includes a plurality of to-be-processed sub-instructions, and the updating of the to-be-processed rendering instruction to a target rendering instruction includes: Determine a first target object and a second target object in the frame to be processed; the first target object is a non-critical object in the picture of the frame to be processed, and the second target object is the same object in the picture of the frame to be processed; Perform instruction clipping on the to-be-processed sub-instruction corresponding to the first target object to obtain a first target to-be-processed sub-instruction; Merging the to-be-processed sub-instructions corresponding to the second target object to obtain a second target to-be-processed sub-instruction; According to the first target to-be-processed sub-instruction and the second target to-be-processed sub-instruction, the to-be-processed rendering instruction is updated to a target rendering instruction.

2. The image rendering method according to claim 1, characterized in that: The step of determining a predicted load state of a graphics processor according to the rendering instruction to be processed includes: Obtaining a first parameter corresponding to the rendering instruction to be processed; The predicted load state of the graphics processor is determined according to the first parameter and the threshold condition.

3. The image rendering method according to claim 2, characterized in that: Before determining the predicted load state of the graphics processor according to the to-be-processed rendering instruction, the method further includes: Get the preset frame; Generate a preset rendering instruction corresponding to the preset frame; Obtaining a second parameter corresponding to the preset rendering instruction; A threshold condition is determined based on the second parameter.

4. The image rendering method according to claim 1, characterized in that: The step of performing instruction clipping on the to-be-processed sub-instruction corresponding to the first target object to obtain the first target to-be-processed sub-instruction includes: Determining initial vertex data corresponding to the first target object; Determining target vertex data from the initial vertex data by a clustering algorithm; The to-be-processed sub-instruction corresponding to the first target object is updated according to the target vertex data to obtain a first target to-be-processed sub-instruction.

5. The image rendering method according to claim 1, characterized in that: The step of merging the to-be-processed sub-instructions corresponding to the second target object to obtain the second target to-be-processed sub-instructions includes: Obtain the to-be-processed sub-instruction corresponding to any second target sub-object; Generate multi-instance rendering instructions corresponding to the multiple second target sub-objects according to the sub-instruction to be processed and the multiple identical second target sub-objects; The multi-instance rendering instruction is determined as a second target sub-instruction to be processed.

6. The image rendering method according to claim 1, characterized in that: After updating the pending rendering instruction to a target rendering instruction when the predicted load state of the graphics processor is a high load state, the method further includes: The operating frequency of the graphics processor is adjusted to a target operating frequency.

7. An image rendering device, characterized in that: include: An acquisition unit, used for acquiring a pending rendering instruction corresponding to a pending frame; A determination unit, configured to determine a predicted load state of a graphics processor according to the rendering instruction to be processed; An updating unit, configured to update the to-be-processed rendering instruction to a target rendering instruction when the predicted load state of the graphics processor is a high load state; A rendering unit, configured to render the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image; The rendering instruction to be processed includes a plurality of sub-instructions to be processed, and the updating unit is specifically used to: determine a first target object and a second target object in the frame to be processed; the first target object is a non-critical object in the picture of the frame to be processed, and the second target object is the same object in the picture of the frame to be processed; The pending sub-instructions corresponding to the first target object are trimmed to obtain first target pending sub-instructions; the pending sub-instructions corresponding to the second target object are merged to obtain second target pending sub-instructions; and according to the first target pending sub-instructions and the second target pending sub-instructions, the pending rendering instructions are updated to target rendering instructions.

8. A computer device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the image rendering method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image rendering method according to any one of claims 1 to 6.

Images