A graphics rendering method and apparatus

Abstract

The application provides a graphics rendering method and device, which is used for reducing the required computing power by means of interval high-low rendering, and improving the definition of a target rendering image by reusing high-definition information of a historical rendering image, so that a device with low computing power can also obtain a high-definition rendering image. The method comprises: rendering first scene data to obtain a first image, the rendering mode comprising a first rendering mode and / or a second rendering mode, the resolution of the image obtained by the first rendering mode being higher than the resolution of the image obtained by the second rendering mode; when the rendering mode of the first scene data comprises the second rendering mode, upsampling the first image to obtain an upsampled image; projecting a target object in a historical rendering image into the first image to obtain a projection frame, the resolution of the target object in the historical rendering image being higher than the resolution of the target object in the first image; and fusing the upsampled image and the projection frame to obtain a target rendering image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image rendering, in particular to a graphics rendering method and device. BACKGROUND

[0002] In the field of rendering, auxiliary information required by up-sampling algorithm calculation can be easily provided in the graphics pipeline, such as motion vector (mv) of a pixel point and depth information of the pixel point, while in the field of ordinary images, the auxiliary information can usually be estimated by algorithm or collected with the assistance of additional sensor devices.

[0003] For example, the current GPU hardware can support a programmable rendering pipeline, which usually includes stages such as vertex shading, shape (primitive) assembly, geometry shading, rasterization, fragment shading, testing and blending, wherein the three stages of vertex shading, geometry shading and fragment shading support user customization through programming. Common vertex shading is generally used for coordinate transformation of model vertices, and fragment shading is used for color calculation of the final displayed pixel points. However, as the resolution of the image increases, the required computing power for rendering also increases exponentially, and therefore, how to reduce the required computing power for rendering becomes a problem to be solved. SUMMARY

[0004] The present application provides a graphics rendering method and device, which reduces the required computing power for rendering by using an interval high-low rendering method, and improves the clarity of the target rendering image by reusing the high-definition information of the historical rendering image, so that a device with lower computing power can also obtain a high-definition rendering image.

[0005] Therefore, in a first aspect, the present application provides a graphics rendering method, comprising: rendering first scene data to obtain a first image, the first scene data being obtained according to a view angle region of a virtual camera, the rendering method including at least one of a first rendering method or a second rendering method, the resolution of the image obtained by the first rendering method being higher than the resolution of the image obtained by the second rendering method, which can be understood as that the first rendering method is a high-definition rendering method, and the second rendering method is a low-definition rendering method; when the rendering method of the first scene data includes the second rendering method, up-sampling the first image to obtain an up-sampled image; obtaining a historical rendering image, the historical rendering image being obtained by rendering second scene data, the target object existing in both the historical rendering image and the first image, and the resolution of the target object in the historical rendering image being higher than the resolution of the target object in the first image; projecting the target object in the historical rendering image into the first image to obtain a projection frame; and fusing the up-sampled image and the projection frame to obtain a target rendering image.

[0006] Therefore, in the embodiments of the present application, at least one of the first rendering mode or the second rendering mode can be selected to render the current view region, if the rendering mode of the current view region includes the second rendering mode, i.e., the resolution of the obtained rendering image is low, the first image obtained by the second rendering mode can be up-sampled to obtain an up-sampled image, so as to improve the resolution of the rendered image. Then, the high-definition object in the historical rendering image is projected into the first image, so as to improve the projection frame with high-definition information and matching the object of the first image, and then the up-sampled image and the projection frame are fused, so as to obtain the high-definition target rendering image. That is, when low-definition rendering is performed, the high-definition information in the high-definition rendering image obtained by rendering can be reused to supplement the details of the low-resolution image obtained by the second rendering mode, so as to obtain a high-definition rendering image. Moreover, the required computing power of the second rendering mode is lower than that of the first rendering mode, so that the method provided by the present application can be deployed in a device with low computing power, and the device with low computing power can also obtain a high-definition rendering image by using the method provided by the present application.

[0007] In a possible implementation, the foregoing rendering of the first scene data to obtain the first image can include: rendering data corresponding to a first ROI region of interest in the first scene data using the second rendering mode to obtain the first image, the first ROI region being a preset region or a region determined from a view region corresponding to the first scene data according to a preset manner.

[0008] Therefore, in the embodiments of the present application, when low-definition rendering is performed, the region of interest (ROI) region in the first scene data can be rendered in low definition. Generally, the model or lighting condition of the ROI region is relatively complex, and low-definition rendering of the ROI region can significantly reduce the required computing power, so that the method provided by the present application can be applied to a device with low computing power.

[0009] In a possible implementation, the foregoing projecting the object in the historical rendering image into the first image to obtain the projection frame can include: regarding a region including a target object in the historical rendering image as a second ROI region; projecting the target object in the second ROI region in the historical rendering image into the first image to obtain the projection frame; and fusing the second ROI region in the projection frame and the first image to obtain the target rendering image.

[0010] Corresponding to the foregoing embodiments, if the ROI region in the current view region is rendered in low definition, when the historical rendering image is projected, the ROI region in the historical rendering image can also be projected, so as to facilitate subsequent use of the high-definition information of the ROI region in the historical rendering image to supplement the details of the low-definition information of the ROI region in the rendering image of the current view region, thereby improving the definition of the ROI region in the current view region and obtaining a high-definition image.

[0011] In a possible implementation, the foregoing rendering in the current view region can further include rendering data of a background region in the first scene data by the first rendering manner to obtain a background rendering image, the background region being a region in the view region corresponding to the first scene data except the first ROI region; and the foregoing method can further include fusing the target rendering image and the background rendering image to obtain an updated target rendering image.

[0012] Therefore, in the embodiments of the application, in addition to the ROI region, the background region is also rendered in high definition, and the high-definition rendering of the background region requires relatively low computing power, so that even if the background part is rendered in high definition, a device with relatively low computing power can be adapted, and the rendering of the background part also needs to fuse the rendering of the background part and the ROI region to form a complete rendering image and obtain a high-definition rendering image.

[0013] In a possible implementation, the foregoing method can further include: if the first scene data meets a preset condition, determining that the rendering manner of the first scene data includes the first rendering manner, the preset condition including one or more of the following: background switching between the first scene data and an image obtained by last rendering; or a motion vector of at least one object in the first scene data being higher than a preset motion value, the motion vector including an offset of the at least one object in the first scene data and the second scene data; or a change in illumination between the first scene data and the second scene data exceeding a preset change amount, the information of the illumination including at least one of illumination intensity, light source type, light source incident direction or light source number; or an interval of N frames between the second scene data and a region rendered by using the first rendering manner last time, N being a positive integer.

[0014] Therefore, in the embodiments of the application, the high-definition rendering or the low-definition rendering can be determined from the background switching, the motion vector, the illumination change or the interval of N frames, so that a rendering manner suitable for a scene can be selected, different three-dimensional scenes can be adapted, and a high-definition rendering image can also be obtained for different three-dimensional scenes, so that low-definition rendering is selected in a suitable scene to reduce the computing power requirement of a device.

[0015] In a possible implementation, the first ROI region is determined based on one or more of information of illumination, information of shadow, information of illumination reflection, or material of the target object of the first scene data. Therefore, in the embodiments of the present application, when rendering the ROI region, a more complex object can be selected based on information of illumination, shadow, illumination reflection, or material of the object, and generally, these objects require more computing power if high-definition rendering is used. Therefore, the present application performs low-definition rendering on these objects, which can significantly reduce the computing power requirement, thereby improving the possibility of deploying the method provided by the present application to a terminal with lower computing power.

[0016] In a possible implementation, the foregoing fusing the up-sampled image and the projection frame to obtain the target rendered image can include: obtaining, by a weight coefficient network, a first weight corresponding to each pixel point in the first image and a second weight corresponding to each pixel point in the projection frame, the weight coefficient network being a neural network for calculating weights corresponding to at least two input frames of images; and fusing the up-sampled image and the projection frame based on the first weight and the second weight to obtain the target rendered image.

[0017] In the embodiments of the present application, the up-sampled image and the projection frame can be fused by a neural network, and generally, the neural network can be implemented by a network processing unit (NPU) in the device, while rendering is generally performed by a graphics processing unit (GPU). Therefore, the present application implements image fusion by the NPU, which can reduce the calculation amount of the GPU, further reduce the computing power requirement of the GPU, thereby improving the rendering efficiency and improving the possibility of deploying the method provided by the present application to a terminal with lower computing power.

[0018] In a possible implementation, the foregoing fusing the up-sampled image and the projection frame based on the first weight and the second weight to obtain the target rendered image can include: interpolating the first weight to obtain a third weight corresponding to each pixel point in the up-sampled image; and fusing the up-sampled image and the projection frame based on the third weight and the second weight to obtain the target rendered image. In the embodiments of the present application, the resolution of the first image is low, or the images input to the weight coefficient network can be down-sampled, and therefore the resolution of the obtained first weight is also low. Therefore, the first weight can be interpolated to obtain a weight with higher resolution, and the up-sampled image and the projection frame can be fused based on the weight with higher resolution, thereby obtaining a high-definition target rendered image.

[0019] In a possible implementation, before fusing the up-sampled image and the projection frame to obtain the target rendering image, the method can further include: correcting the projection frame by using a correction network to obtain a corrected projection frame, the correction network being a neural network used for filtering an input image.

[0020] In the implementation of the present application, after the projection using the historical rendering image, there can be occlusion between objects or light reversal error, and therefore the projection frame can be corrected to reduce ghosting in the projection frame and obtain a corrected projection frame. In addition, the present application can correct the projection frame by using a correction network, so that the correction can be implemented by using an NPU to reduce the demand for GPU computing power, thereby improving the possibility of deploying the method provided by the present application to a terminal with low computing power.

[0021] In a possible implementation, before rendering the first scene data to obtain the first image, the method further includes: rendering second scene data by using a first rendering manner to obtain a historical rendering image.

[0022] Therefore, in the implementation of the present application, the second scene data can be rendered in high definition to obtain a high-definition historical rendering image.

[0023] In a possible implementation, if the ROI region in the current view region includes a moving target object, projecting the object in the historical rendering image into the first image includes: projecting the target object in the historical rendering image into the first image according to a motion vector of the target object to obtain a projection frame, the motion vector including an offset of the target object between the first scene data and the second scene data.

[0024] In the implementation of the present application, the object in the historical rendering image can be projected according to the motion vector of the object, so that the object in the historical rendering image can be more accurately projected, the ghosting in the projection frame can be reduced, and the definition of the projection frame can be improved.

[0025] In a possible implementation, the foregoing projecting the target object in the historical rendering image into the first image according to the motion vector of the target object to obtain a projection frame can include: determining a position of the target object in the historical rendering image in the first image according to the motion vector of the target object, and assigning a value to the position to obtain the projection frame.

[0026] In the implementation of the present application, the position of the object in the historical rendering image in the first image can be determined according to the motion vector of the object, so that the corresponding position in the first image is assigned a value according to the pixel value or color value of the object in the historical rendering image, thereby obtaining a high-definition projection frame.

[0027] In a second aspect, the present application provides a terminal, comprising a GPU and an NPU, and the GPU and the NPU are configured to perform the following steps:

[0028] The GPU is configured to render the first scene data to obtain a first image, the first scene data is obtained according to a perspective region of a virtual camera, and the rendering manner comprises at least one of a first rendering manner or a second rendering manner, and a resolution of an image obtained by the first rendering manner is higher than a resolution of an image obtained by the second rendering manner.

[0029] The NPU is configured to, when the rendering manner of the first scene data comprises the second rendering manner, up-sample the first image to obtain an up-sampled image.

[0030] The GPU is further configured to obtain a historical rendering image, the historical rendering image is obtained by rendering second scene data, a target object exists in the historical rendering image and the first image, and a resolution of the target object in the historical rendering image is higher than a resolution of the target object in the first image.

[0031] The GPU is further configured to project the target object in the historical rendering image into the first image to obtain a projection frame.

[0032] The NPU is further configured to fuse the up-sampled image and the projection frame to obtain a target rendering image.

[0033] Effects of the second aspect and any possible implementation manner of the present application can refer to the effects of the first aspect and any possible implementation manner of the present application, which will not be described here.

[0034] In a possible implementation manner, the GPU is further configured to render data corresponding to a first ROI region of interest in the first scene data by using the second rendering manner to obtain the first image, the first ROI region is a preset region or a region determined from a perspective region corresponding to the first scene data according to a preset manner.

[0035] In a possible implementation manner, the GPU is further configured to: take a region including the target object in the historical rendering image as a second ROI region; and project the target object in the second ROI region in the historical rendering image into the first image to obtain the projection frame.

[0036] The NPU is specifically configured to fuse the second ROI region in the projection frame and the first image to obtain the target rendering image.

[0037] In a possible implementation manner, the GPU is further configured to: render data of a background region in the first scene data by using the first rendering manner to obtain a background rendering image, the background region is a region in the perspective region corresponding to the first scene data except the first ROI region; and fuse the target rendering image and the background rendering image to obtain an updated target rendering image.

[0038] In a possible implementation, if the current view region meets a preset condition, the GPU is further configured to determine that the rendering manner of the current view region comprises the first rendering manner, and the preset condition comprises one or more of the following:

[0039] a background switch between the first scene data and an image obtained by last rendering; or a motion vector of at least one object in the first scene data is higher than a preset motion value, the motion vector comprising an offset of the at least one object in the first scene data and the second scene data; or a change of illumination between the first scene data and the second scene data exceeds a preset change amount, the information of the illumination comprising at least one of illumination intensity, light source type, light source incident direction or light source number; or the second scene data and a region rendered by using the first rendering manner last time are separated by N frames, N being a positive integer.

[0040] In a possible implementation, the first ROI region is determined based on one or more of information of illumination, information of shadow, information of illumination reflection or material of a target object in the first scene data.

[0041] In a possible implementation, the NPU is further configured to: obtain, by using a weight coefficient network, first weights corresponding to each pixel point in the first image and second weights corresponding to each pixel point in the projection frame, the weight coefficient network being a neural network configured to calculate weights corresponding to at least two input images; and fuse the up-sampling image and the projection frame based on the first weights and the second weights to obtain the target rendering image.

[0042] In a possible implementation, the NPU is further configured to, before fusing the up-sampling image and the projection frame to obtain the target rendering image, correct the projection frame by using a correction network to obtain a corrected projection frame, the correction network being a neural network configured to filter an input image.

[0043] In a possible implementation, the GPU is further configured to, before rendering the first scene data to obtain the first image, render the second scene data by using the first rendering manner to obtain a historical rendering image.

[0044] In a third aspect, an embodiment of the present application provides a graphics rendering apparatus having a function of implementing the image processing method in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0045] In a fourth aspect, an embodiment of the present application provides a graphics rendering apparatus, comprising a processor and a memory, wherein the processor and the memory are interconnected through a circuit, and the processor invokes program codes in the memory to execute functions related to processing in the graphics rendering method according to any one of the first aspect.

[0046] In a fifth aspect, an embodiment of the present application provides a graphics rendering apparatus, which can also be referred to as a digital processing chip or a chip. The chip comprises a processing unit and a communication interface. The processing unit acquires program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is configured to execute functions related to processing in the graphics rendering method according to the first aspect or any one of the optional implementation manners of the first aspect.

[0047] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium comprising instructions, which, when executed on a computer, cause the computer to execute the method according to the first aspect or any one of the optional implementation manners of the first aspect.

[0048] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to execute the method according to the first aspect or any one of the optional implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A schematic diagram of an artificial intelligence subject framework for application of the present application;

[0050] Figure 2A A schematic diagram of a system architecture provided by an embodiment of the present application;

[0051] Figure 2B A schematic diagram of a structure of an electronic device provided by an embodiment of the present application;

[0052] Figure 3 A flowchart of a graphics rendering method provided by the present application;

[0053] Figure 4 A flowchart of another graphics rendering method provided by the present application;

[0054] Figure 5 A schematic diagram of a view angle region provided by the present application;

[0055] Figure 6 A schematic diagram of a projection mode provided by the present application;

[0056] Figure 7 A schematic diagram of a projection frame provided by the present application;

[0057] Figure 8A projection frame correction method provided in the present application is shown in the figure;

[0058] Figure 9 A fusion weight calculation method provided in the present application is shown in the figure;

[0059] Figure 10 A difference between low definition rendering and high definition rendering provided in the present application is shown in the figure;

[0060] Figure 11 An image after rendering provided in the present application is shown in the figure;

[0061] Figure 12 A flow chart of another image rendering method provided in the present application is shown in the figure;

[0062] Figure 13 An ROI region provided in the present application is shown in the figure;

[0063] Figure 14 A low definition rendering image of an ROI region provided in the present application is shown in the figure;

[0064] Figure 15 A rendering image of an ROI region in a current view region provided in the present application is shown in the figure;

[0065] Figure 16 An output image provided in the present application is shown in the figure;

[0066] Figure 17 A rendering effect of an ROI region in a scheme provided in the present application and a commonly used scheme is shown in the figure;

[0067] Figure 18 A structure of a terminal provided in the present application is shown in the figure;

[0068] Figure 19 A structure of another terminal provided in the present application is shown in the figure;

[0069] Figure 20 A structure of an image rendering device provided in the present application is shown in the figure;

[0070] Figure 21 A structure of another image rendering device provided in the present application is shown in the figure. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described below with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0072] First, the overall workflow of artificial intelligence systems is described, please see Figure 1 , Figure 1 The structure of the artificial intelligence subject framework is shown, and the following describes the artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0073] (1) Infrastructure

[0074] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), or field programmable gate array (FPGA) hardware acceleration chips); The underlying platform includes distributed computing framework and related platform support and support for networks, which can include cloud storage and computing, interconnection networks, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for computing.

[0075] (2) Data

[0076] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, voice, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0077] (3) Data processing

[0078] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0079] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. of intelligent information symbolized and formalized by data.

[0080] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to solve problems according to reasoning control strategies, and the typical functions are search and matching.

[0081] Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0082] (4) General ability

[0083] After the data is processed as mentioned above, some general abilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0084] (5) Intelligent products and industry applications

[0085] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of overall solutions of artificial intelligence, and realize the application of intelligent information decision-making products. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0086] The embodiments of the present application relate to a large number of neural networks and image-related applications. In order to better understand the scheme of the embodiments of the present application, the following will first introduce the related terms and concepts in the field of neural networks and images that may be involved in the embodiments of the present application.

[0087] (1) Neural network

[0088] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs and intercept 1 as input. The output of the operation unit can be as shown in the following formula:

[0089]

[0090] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0091] (2) Deep neural network

[0092] Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of intermediate layers is the intermediate layer, or called the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0093] Although DNN looks very complex, each layer can be represented as a linear relationship expression: where, is the input vector, is the output vector, is the offset vector or called bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is only a simple operation on the input vector to obtain the output vector Due to the large number of layers in DNN, the number of coefficients W and offset vectors is also relatively large. These parameters are defined in the DNN as follows: taking the coefficient w as an example: assuming in a three-layer DNN, the linear coefficient from the fourth neuron of the second layer to the second neuron of the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

[0094] In summary, the coefficient from the k-th neuron of the L-1-th layer to the j-th neuron of the L-th layer is defined as

[0095] It is noted that the input layer is without W parameters. In deep neural networks, more intermediate layers allow the network to better capture the complexity of real-world situations. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity" to perform more complex learning tasks. Training a deep neural network is the process of learning the weight matrices, and the ultimate goal is to obtain the trained weight matrices of all layers of the deep neural network (weight matrices formed by vectors W of many layers).

[0096] (3) Convolutional neural network

[0097] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The CNN contains a feature extractor composed of convolutional layers and sub-sampling layers, which can be regarded as a filter. The convolutional layer refers to the neuron layer in the CNN that performs convolution processing on the input signal. In the convolutional layer of the CNN, a neuron can be connected only to part of the adjacent layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights are the convolution kernel. The shared weights can be understood as the way of extracting image information being independent of the position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the CNN. In addition, the direct benefit of shared weights is to reduce the connections between layers of the CNN and to reduce the risk of overfitting.

[0098] (4) Loss function

[0099] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is actually intended to be predicted, the weight vectors of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the target value that is actually intended to be predicted (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vectors are adjusted to make the prediction lower, and the adjustment is continuously made until the deep neural network can predict the target value that is actually intended to be predicted or a value very close to the target value that is actually intended to be predicted. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss. The loss function can generally include error square mean square, cross-entropy, logarithm, exponential, etc. For example, the error mean square can be used as the loss function, which is defined as Specifically, a specific loss function can be selected according to an actual application scenario.

[0100] (5) Back propagation algorithm

[0101] The neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output produces an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0102] (6) Rendering

[0103] In an electronic device with display function, the process of converting 3D / 2D model into display image is widely used in the fields of games, film special effects, etc. The rendering process generally includes modeling, material construction, animation construction, and rendering display.

[0104] (7) Deep learning super sampling (DLSS)

[0105] It mainly refers to using deep learning to up-sample the rendered image to reduce the time-consuming of rendering shading.

[0106] (8)Temporal Anti-Aliasing (TAA)

[0107] TAA is a widely used anti-aliasing algorithm in commercial game engines in recent years, which is a rendering field based on temporal multiplexing framework. The post-processing method is integrated into the rendering process. The existing DLSS technology and the ray tracing denoising algorithm SVGF, BMFR are also based on the improvement technology of TAA.

[0108] (9)Physic Based Rendering (PBR)

[0109] It refers to a collection of rendering technologies based on different degrees of basic theories that are more consistent with the physical principles of the real world. The core is to use a way that is more consistent with the physical law to simulate the light of the object surface. This way is more realistic than the traditional light algorithm based on Phong light model or Bilnn-Phong light model. And the form of PBR is closer to the physical properties, so the surface material can be adjusted based on the physical parameters to make the lighting effect more normal.

[0110] (10)Forward Rendering

[0111] Also known as forward shading, it traverses all light sources for each object in the scene to calculate the final display color.

[0112] (11)Deferred Rendering

[0113] It is not proposed to solve the problem that the forward rendering shading calculation time is linearly related to the number of light sources and objects. In the rendering process, all geometric information required for lighting shading calculation of objects to be rendered is first rendered to a few maps (G-buffer), and then the final lighting color is calculated by traversing the pixel points of G-buffer. It is suitable for large number of rendering objects and multi-light source scene.

[0114] (12)Multisampling Anti-Aliasing (MSAA)

[0115] By calculating the color of multiple sub-pixels for each pixel point, the final pixel color to be displayed is synthesized. This function is generally supported by hardware, suitable for forward rendering pipeline, and can better handle geometric aliasing.

[0116] (13)Mask

[0117] The mask can be understood as data similar to an image. In the embodiments of the present application, the mask can be fused with the image, so that the attention of part of the content in the image is higher. Generally, the mask can be used to extract a region of interest (ROI). For example, a pre-prepared mask of a region of interest is fused with a to-be-processed image to obtain a region of interest image. The image values in the region of interest remain unchanged, and the image values outside the region are all 0. The mask can also have a shielding effect. The mask can shield some regions of the image, so that the regions do not participate in processing or calculation of processing parameters, or only the shielded regions are processed or counted.

[0118] (14) Foreground, background

[0119] Generally, the foreground can be understood as a subject included in an image or an object that needs to be paid attention to. In the embodiments of the present application, the object in the foreground of the image is referred to as a foreground object. The foreground can also be understood as a region of interest (ROI) in the image. The background is a region other than the foreground in the image. For example, an image including a traffic signal lamp, the foreground (or referred to as a foreground object) in the image is the region where the traffic signal lamp is located, and the background is the region other than the foreground in the image.

[0120] The method for rendering a graph provided in the embodiments of the present application can be executed on a server, and can also be executed on a terminal device. Correspondingly, the neural network mentioned below in the present application can be deployed on a server or on a terminal. The specific deployment can be adjusted according to the actual application scenario. For example, the method for rendering a graph provided in the present application can be deployed in a terminal in the form of a plug-in. The terminal device can be a mobile phone, a tablet personal computer (TPC), a media player, a smart television, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smart watch, a wearable device (WD), or an autonomous vehicle, and the like, which have an image processing function. The embodiments of the present application do not limit this. The following exemplary description is exemplarily taken with the method for rendering a graph provided in the present application being deployed in a terminal.

[0121] All or part of the processes in the method for rendering a graph provided in the present application can be implemented by a neural network, such as the upsampling, projection, or fusion steps. Generally, the neural network needs to be deployed on a terminal after training, such as Figure 2AAs shown, the embodiment of the present application provides a system architecture 100. In Figure 2A In the embodiment, the data collection device 160 is configured to collect training data. In some optional implementations, for graphics rendering, the training data can include a large number of rendered high-quality images and unrendered three-dimensional models, etc.

[0122] After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130. Optionally, the training set mentioned in the following embodiments of the present application can be obtained from the database 130 or by inputting data by a user.

[0123] The target model / rule 101 can be a neural network trained in the embodiment of the present application, which can include one or more networks for calculating fusion weights or correcting projected images, etc.

[0124] The training device 120 based on the training data to obtain the target model / rule 101 is described below. The training device 120 processes the input three-dimensional model, compares the output image with the high-quality rendered image corresponding to the input three-dimensional model, and until the difference between the output image and the high-quality rendered image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0125] The target model / rule 101 described above can be used to implement a neural network trained by the graphics rendering method of the embodiment of the present application, that is, the target model / rule 101 can be used to obtain the processing result after the to-be-processed data (such as an image to be rendered or an image that has been rendered and needs to be further processed) is inputted into the target model / rule 101 after relevant preprocessing. The target model / rule 101 in the embodiment of the present application can be a neural network mentioned below, which can be a CNN, a DNN or an RNN type neural network. It should be noted that in actual applications, the training data maintained in the database 130 can not all come from the collection of the data collection device 160, but can also be received from other devices. In addition, it should be noted that the training device 120 can not completely train the target model / rule 101 based on the training data maintained in the database 130, but can also obtain training data from the cloud or other places for model training, which is not limited in the present application.

[0126] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as being applied to Figure 2AThe execution device 110 shown, which can also be referred to as a computing device, can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) terminal, a vehicle-mounted terminal, etc., and can also be a server or a cloud device, etc. Figure 2A In some embodiments, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. A user can input data to the I / O interface 112 through a client device 140, and the input data can include, in some embodiments, data to be processed input by the client device. The client can be another hardware device such as a terminal or a server, etc., or can be software deployed on a terminal such as an APP, a web page, etc.

[0127] The preprocessing modules 113 and 114 are configured to perform preprocessing on the input data (e.g., data to be processed) received by the I / O interface 112. In some embodiments, the preprocessing modules 113 and 114 can not be used (only one of them can be used), and the input data can be directly processed by the computing module 111.

[0128] During the preprocessing of the input data by the execution device 110 or during the execution of the computing module 111 of the execution device 110, the execution device 110 can call data, codes, etc. in the data storage system 150 for corresponding processing, and can store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0129] Finally, the I / O interface 112 returns the processing result to the client device 140, thereby providing the user with the result. For example, if the first neural network is used for image classification, the processing result is a classification result, and the I / O interface 112 returns the classification result obtained above to the client device 140, thereby providing the user with the result.

[0130] It should be noted that the training device 120 can generate a corresponding target model / rule 101 based on different training data for different targets or tasks, and the corresponding target model / rule 101 can be used to achieve the above target or complete the above task, thereby providing the user with the required result. In some scenarios, the execution device 110 and the training device 120 can be the same device or located in the same computing device. For ease of understanding, the execution device and the training device are introduced separately in this application and are not limited.

[0131] In some embodiments, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. A user can input data to the I / O interface 112 through a client device 140, and the input data can include, in some embodiments, data to be processed input by the client device. The client can be another hardware device such as a terminal or a server, etc., or can be software deployed on a terminal such as an APP, a web page, etc. Figure 2AIn the case shown, the user can manually give input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112, and if the user's authorization is required to enable the client device 140 to automatically send input data, the user can set the corresponding permission in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 140 can also act as a data collection end to collect the input data input into the I / O interface 112 and the predicted label output by the I / O interface 112 as new sample data and store them into the database 130. Of course, the collection can also be performed without the client device 140, and the I / O interface 112 can directly store the input data input into the I / O interface 112 and the predicted label output by the I / O interface 112 as new sample data into the database 130.

[0132] It should be noted that, Figure 2A The system architecture provided in the embodiments of the present application is only a schematic diagram, and the positional relationship between the devices, components, modules, etc. shown in the diagram does not constitute any limitation. For example, in Figure 2A In the case shown, the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can also be placed in the execution device 110.

[0133] As shown in Figure 2A According to the training device 120, a target model / rule 101 is trained, which can be a neural network in the present application in the embodiments of the present application. Specifically, the neural network provided in the embodiments of the present application can include a CNN, a deep convolutional neural network (DCNN), a recurrent neural network (RNN), or a neural network constructed.

[0134] The graphics rendering method in this embodiment can be executed by an electronic device, namely the aforementioned execution device. This electronic device includes a CPU and a GPU, capable of rendering images. Of course, other devices may also be included, such as an NPU or ASIC, etc., which are merely illustrative examples and will not be elaborated further. For example, the electronic device may be a mobile phone, tablet computer, laptop computer, PC, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless electronic device in industrial control, wireless electronic device in self-driving, wireless electronic device in remote medical surgery, wireless electronic device in smart grid, wireless electronic device in transportation safety, wireless electronic device in smart city, wireless electronic device in smart home, etc. The electronic device may be a device running Android, iOS, Windows, or other systems. This electronic device can run applications that require rendering 3D scenes to obtain 2D images, such as game applications, lock screen applications, map applications, or monitoring applications.

[0135] To facilitate understanding, the following will be combined with... Figure 2B A detailed description of the specific structure of the electronic device is provided. Please refer to [reference needed]. Figure 2B , Figure 2B This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0136] In one possible embodiment, such as Figure 2B As shown, the electronic device 2000 may include: a central processing unit 2001, a graphics processor 2002, a display device 2003, and a memory 2004. Optionally, the electronic device 2000 may further include at least one communication bus. Figure 2B (Not shown in the image) is used to enable communication between various components.

[0137] It should be understood that various components in the electronic device 2000 can also be coupled through other connectors, which can include various types of interfaces, transmission lines, buses, and the like. The various components in the electronic device 2000 can also be connected in a radial manner around the central processor 2001. In various embodiments of the present application, coupling refers to electrical connection or communication with each other, including direct connection or indirect connection through other devices.

[0138] The connection mode of the central processor 2001 and the graphics processor 2002 can also be various, and is not limited to the mode shown. Figure 2B The central processor 2001 and the graphics processor 2002 in the electronic device 2000 can be located on the same chip, or can be independent chips.

[0139] The functions of the central processor 2001, the graphics processor 2002, the display device 2003, and the memory 2004 will be briefly introduced below.

[0140] The central processor 2001 is used to run the operating system 2005 and the application program 2006. The application program 2006 can be a graphics application program, such as a game, a video player, and the like. The operating system 2005 provides a system graphics library interface, and the application program 2006 generates an instruction stream for rendering a graphics or image frame and the required related rendering data through the system graphics library interface and the driver provided by the operating system 2005, such as a graphics library user mode driver and / or a graphics library kernel mode driver. The system graphics library includes but is not limited to: an open graphics library for embedded system (OpenGL ES), a Khronos platform graphics interface, or a Vulkan (a cross-platform drawing application program interface), and the like. The instruction stream contains a series of instructions, which are usually call instructions for the system graphics library interface.

[0141] Optionally, the central processor 2001 can include at least one of the following types of processors: an application processor, one or more microprocessors, a digital signal processor (DSP), a microcontroller unit (MCU), an artificial intelligence processor, and the like.

[0142] The central processor 2001 can further include necessary hardware accelerators, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an integrated circuit for implementing logical operations. The processor 2001 can be coupled to one or more data buses for transmitting data and instructions between various components of the electronic device 2000.

[0143] The graphics processor 2002 is configured to receive a stream of graphics instructions sent by the processor 2001, generate a rendering target through a rendering pipeline, and display the rendering target to the display device 2003 through a layer compositing display module of the operating system. The rendering pipeline can also be referred to as a rendering pipeline, a pixel pipeline, or a pixel pipeline, which is a parallel processing unit inside the graphics processor 2002 for processing graphics signals. Multiple rendering pipelines can be included in the graphics processor 2002, and the multiple rendering pipelines can independently process graphics signals in parallel. For example, the rendering pipeline can perform a series of operations in the process of rendering a graphics or image frame, and typical operations can include vertex processing, primitive processing, rasterization, fragment processing, and the like.

[0144] Optionally, the graphics processor 2002 can include a general-purpose graphics processor that executes software, such as a GPU or other types of specialized graphics processing units.

[0145] The display device 2003 is configured to display various images generated by the electronic device 2000, which can be a graphical user interface (GUI) of the operating system or image data processed by the graphics processor 2002, including still images and video data.

[0146] Optionally, the display device 2003 can include any suitable type of display screen, such as a liquid crystal display (LCD) or a plasma display or an organic light-emitting diode (OLED) display, and the like.

[0147] The memory 2004 is a transmission channel between the central processor 2001 and the graphics processor 2002, and can be a double data rate synchronous dynamic random access memory (DDR SDRAM) or other type of cache.

[0148] Commonly used rendering up-sampling is similar to common image and video super-resolution, and is proposed to solve the problem of reduced temporal effect caused by insufficient sampling. The difference is that rendering is a discrete sample point in space and time, and low-resolution rendering will cause high overlap or distortion, so the up-sampling algorithm in the rendering pipeline is usually an anti-aliasing and interpolation algorithm; and the data source in the ordinary image and video field is mostly from the camera, and the color of each pixel point is the integral of a pixel area. The problem of insufficient low-resolution sampling is blurring, and the up-sampling method is deblurring and interpolation algorithm. For example, the screen resolution of the current mobile terminal (such as a mobile phone) is usually 1080p, and 2K screens and 4K screens will also gradually appear. Higher rendering resolution means greater GPU computing load. Usually, the time consumption of the geometry shading stage accounts for 20% of the total time consumption, and the time consumption of the fragment shading stage accounts for 80% of the total time consumption. Theoretically, the load increases by 4 times when rendering 540p and rendering 1080p, and the actual test is about 3.4 times. Theoretically, the load increases by 1.8 times when rendering 1080p and rendering 2k, and the actual test is about 1.6 times. At the same time, the fragment shading stage is more sensitive to resolution changes, and reducing the calculation amount of the fragment shading stage can greatly reduce the load of the GPU.

[0149] However, the commonly used time domain anti-aliasing scheme is for PC, which is computationally complex and cannot be applied to mobile terminals.

[0150] Therefore, the present application provides a graphics rendering method, which reduces the computing power required for rendering by performing high-definition rendering and low-definition rendering at intervals, and reuses high-definition information included in the projection frame to obtain a rendering image with higher clarity. The graphics rendering method provided by the present application will be described in detail below.

[0151] First, for the convenience of understanding, some application scenarios of the graphics rendering method provided by the present application are exemplarily introduced.

[0152] For example, in one scenario, the method provided by the present application can be applied in some AR or VR applications, such as shopping, gaming, projection and the like, and the method of rendering graphics provided by the present application can be used by smart wearable devices (such as VR glasses / AR glasses and the like), mobile terminals (such as mobile phones, tablets and the like) or other devices that can carry AR or VR applications and the like. For example, a VR application program can be installed in a wearable device to provide services for users, and the VR application can provide a plurality of three-dimensional scenes for users, and users can watch the three-dimensional scenes in the VR application through the display screen of the wearable device, which is equivalent to providing an immersive experience for users and improving user experience. For the display of these three-dimensional scenes, after the three-dimensional model is constructed, the three-dimensional model needs to be rendered, and the rendered image output by the method of rendering graphics provided by the present application is provided to users in the form of visual experience.

[0153] For another example, in another scenario, the method provided by the present application can be applied in an AR game, and an application program of the AR game can be installed in a mobile terminal or a wearable device of a user, and the user can immerse in the AR game through the mobile terminal or the wearable device to improve user experience. When the mobile terminal or the wearable device runs the AR game, the three-dimensional scene in the game needs to be rendered, so that the three-dimensional scene can be presented in the form of an image, and the three-dimensional scene in the game can be rendered by the method of rendering graphics provided by the present application, so that a high-definition image after rendering is obtained to improve user experience.

[0154] Referring to Figure 3 The flowchart of the method of rendering graphics provided by the present application is as follows.

[0155] 301. Obtain first scene data.

[0156] The first scene data can be three-dimensional scene data, two-dimensional scene data, or scene data with more dimensions, and the first scene data can include a plurality of two-dimensional or three-dimensional models, and each model can be composed of a plurality of basic primitives. The present application exemplarily takes a three-dimensional scene as an example for illustrative description, and the three-dimensional scene or three-dimensional data mentioned below can be replaced by two-dimensional scene or two-dimensional data, which will not be described below.

[0157] Specifically, the first scene data can be determined by the perspective region of a virtual camera, and the present application exemplarily takes the perspective region corresponding to the first scene data as a current perspective region, and takes the perspective region corresponding to the previous frame or multiple frames of second scene data as an adjacent perspective region or the previous perspective region of the virtual camera.

[0158] It can be understood that when rendering, the first scene data can be obtained by taking the data corresponding to the current perspective region from a larger two-dimensional or three-dimensional scene through the perspective region of the virtual camera. The application is exemplarily illustrated by taking the data corresponding to the current perspective region from a three-dimensional scene as an example. The three-dimensional scene data mentioned below can also be replaced by two-dimensional scene data or more dimensional scene data, and the application does not limit this.

[0159] For example, the three-dimensional scene data can be constructed by selecting three-dimensional models from a three-dimensional model library by the server. The three-dimensional models can include models or objects of shapes such as trees, light sources, houses, buildings, geographical environments, characters, or animals. The selected three-dimensional models can constitute a three-dimensional scene. In an actual application scenario, as the perspective of the virtual camera shifts in the three-dimensional scene, the models in the perspective region can also change.

[0160] The three-dimensional scene data can be data in various scenes. Some three-dimensional data in some scenes is exemplarily introduced below. For example, the three-dimensional scene data can include data for constructing a three-dimensional scene in an AR or VR application, such as some AR / VR games, AR maps, and the like. Various three-dimensional models are constituted by basic primitives, and then a virtual three-dimensional scene similar to reality is constituted by the three-dimensional models. The three-dimensional scene can be rendered into a visible image through the following steps, and can be displayed on a display screen of an AR or VR device, so that a user can observe the three-dimensional scene through the display screen, and the user experience is improved.

[0161] Optionally, in a possible implementation, before step 301, the second scene data can also be high-definition rendered to obtain a high-definition historical rendering image. The second scene data can be scene data corresponding to a previous perspective region or a perspective region adjacent to the current perspective region of the virtual camera.

[0162] 302, it is judged whether the first scene data is low-definition rendered. If yes, step 303 is performed. If no, step 307 is performed.

[0163] The rendering mode of the scene data can be divided into high-definition rendering (or referred to as a first rendering mode) and low-definition rendering (or referred to as a second rendering mode). The resolution of the image obtained by high-definition rendering is higher than the resolution of the image obtained by low-definition rendering. Correspondingly, the definition of the image obtained by high-definition rendering is better than the definition of the image obtained by low-definition rendering. The more complex the model is, the more computing resources are consumed for rendering.

[0164] The current view region is a region in the scene that needs to be rendered at present. It can be understood that when rendering, a virtual camera can be determined in advance, and then rendering is performed from the perspective of the virtual camera. For example, when a user uses a game built by a three-dimensional scene, the display perspective of the user can follow the movement of the character controlled by the user, and the perspective of the user is the perspective of the virtual camera, and the visible region displayed in the display is the current view region.

[0165] Specifically, it can be judged whether the first scene data meets a preset condition to determine whether the rendering mode of the first scene data includes high-definition rendering. The preset condition can include one or more of the following, but is not limited to:

[0166] Scene switching, or background switching, between the first scene data and the second scene data, such as when a user enters a copy from a current scene, that is, enters a new scene, it can be considered that scene switching is performed; or the motion vector of the object in the current view region, that is, the object in the first scene data, is higher than a preset motion value, the motion vector includes the offset of the object between the first scene data and the second scene data of the three-dimensional scene, that is, the offset between the current view region and the last view region, such as optical flow or the motion speed of the object, and the motion vector can be understood as the offset value of the pixels between two view regions; or the change of illumination between the current view region and the adjacent view region exceeds a preset change amount, the illumination information includes at least one of illumination intensity, light source type, light source incident direction, or light source number, and generally, illumination change can cause large changes in the three-dimensional model, so high-definition rendering is required to obtain clearer images; or the interval between the current view region and the region last rendered in high definition is N frames, N is a positive integer, and the N can be a preset value or a value determined according to user input data.

[0167] Therefore, in the embodiments of the present application, when scene switching, a large motion vector, a large change in illumination, or an interval of N frames occurs, high-definition rendering is performed, which is equivalent to interval high-definition rendering and low-definition rendering. Some scenes with large changes can use high-definition rendering, while scenes with less change can use low-definition rendering, and then reuse high-definition information in the historical rendering image to improve the clarity of the image output by low-definition rendering, and a high-definition image can also be obtained. By low-definition rendering, the computing resources required for rendering are reduced, so that the method provided by the present application can not only be deployed in a server, but also can be deployed in a device with lower computing power, such as a mobile phone, a television, or a terminal device such as a tablet computer, thereby improving the user experience of the terminal device.

[0168] It should be noted that the manner of rendering the scene can include high-definition rendering and / or low-definition rendering. For the same view angle region, high-definition rendering and low-definition rendering can be performed simultaneously. For example, the foreground part with complex model or lighting conditions can be rendered in low definition to reduce the demand for computing power. The background part with simple model or lighting conditions can be rendered in high definition, and the computing power consumed is also low, so that the rendering effect of the background part is better, and the user experience is improved.

[0169] 303. render the first scene data in low definition to obtain a first image.

[0170] If the manner of rendering the first scene data includes low-definition rendering, the current view angle region can be rendered in low definition to obtain a first image.

[0171] Specifically, the high-definition rendering or low-definition rendering in the embodiment can adopt a rendering manner such as PBR, forward rendering pipeline or deferred rendering pipeline. The difference between high-definition rendering and low-definition rendering is that the resolution of the image output by low-definition rendering is lower. For example, when performing graphic rendering, if low-definition rendering is adopted, the coloring can be performed in units of multiple points to reduce the computing power required for rendering, that is, it is not necessary to calculate the color value of each point, but the color value is calculated in units of multiple points, so that the computing resources required for calculating the color value are reduced, and the rendering efficiency is improved.

[0172] In a possible implementation, the first ROI (region of interest) region in the first scene data can be rendered in low definition to obtain a first image. The first ROI region is a preset region or a region determined according to a preset manner. It can be understood that the current view angle region can be divided into foreground and background parts. The foreground can be regarded as an ROI region. The ROI region in the current view angle region can be rendered in low definition, and the background part can be rendered in high definition. Generally, the background part can have low model complexity or simple lighting conditions, so the computing power required for rendering is also low. The ROI region can have high model complexity or complex lighting conditions, so the computing power required for rendering is large. The application provides a low-definition rendering manner to reduce the demand for computing power, so that the method provided by the application can be deployed in a terminal with low computing power, and the user experience is improved.

[0173] The first ROI region can be determined by referring to information of light, information of shadow, information of light reflection or material of the object in the current view region. The information of light can include light intensity, light source type, light source incident direction or light source number, and the information of shadow can include area of the shadow or number of regions with the shadow.

[0174] Alternatively, the first ROI region can also be a region determined according to user input data. For example, if the method provided in the present application is deployed in a terminal, the user can set the region to be rendered in low resolution through the touch screen of the terminal, so that the user can select the ROI region according to the actual application scenario, and determine which regions to be rendered in low resolution, thereby improving the user experience.

[0175] In addition, if the first ROI region of the first scene data is rendered in low resolution, the background part in the current view region except the first ROI region can also be rendered in high definition, thereby obtaining a rendered image of the high-definition background part. Generally, the calculation amount required for rendering the background part is lower than the calculation amount required for rendering the ROI region, so in the present application, the ROI region can be rendered in low resolution and the background part can be rendered in high definition, only a small amount of calculation is required to obtain the rendered image, so that the scheme provided in the present application can be applied to devices with low computing power, has strong generalization ability, and improves the user experience of users using devices with low computing power.

[0176] 304, up-sampling the first image to obtain an up-sampled image.

[0177] After the current view region is rendered in low resolution to obtain the first image, the resolution of the image output by the low-resolution rendering is low, so the first image can be up-sampled to obtain an up-sampled image with higher resolution.

[0178] Specifically, the up-sampling method can include interpolation or transposed convolution, such as bilinear interpolation, bicubic interpolation and the like. The present application exemplarily takes the up-sampling by interpolation as an example for illustration, and the following mentioned interpolation can be replaced by other up-sampling operations, which will not be described hereinafter.

[0179] 305, projecting the target object in the historical rendered image into the first image to obtain a projection frame.

[0180] In the embodiment, a historical rendering image is also obtained, which can be obtained by rendering the second scene data. The historical rendering image and the first image include the same target object, and the resolution of the target object in the historical rendering image is higher than that in the first image.

[0181] The historical rendering image can be a rendering image corresponding to a neighboring view angle or a previous view angle region of the virtual camera of the current view angle, or an image obtained by rendering a previous frame or multiple frames, and has a higher resolution than the first image, which can be understood as a high-definition rendering image. The target object in the historical rendering image can be projected into the first image to obtain a projection frame. The target object is an object included in both the first image and the historical rendering image.

[0182] Generally, an identifier can be added to an object in the scene data, and each object can have a unique identifier. The common target object included in the first scene data and the second data can be determined by the identifier of the object, and the position and motion state of the target object in each scene can be determined.

[0183] Optionally, the target object in the historical rendering image can be projected into the first image according to a motion vector to obtain a projection frame. The motion vector includes the offset of the target object in the neighboring view angle region and the current view angle region of the three-dimensional scene.

[0184] Specifically, the position of the target object in the historical rendering image in the first image is determined according to the motion vector of the target object, and the position is valued (such as color value, depth value, or value corresponding to different IDs, etc.) to obtain a projection frame. It can be understood that the position of the object moving from the neighboring view angle to the current view angle can be determined according to the motion speed of the object, and then the position in the first image is valued, so that the high-definition object in the historical rendering image can be projected into the first image to obtain a high-definition first image.

[0185] In a possible implementation, a second ROI region can be determined from the historical rendering image, and the second ROI region and the first ROI region described above include the same object. The second ROI region in the historical rendering image can be projected into the first image to obtain a projection frame. Therefore, in the embodiment of the application, if only the first ROI region in the current view angle region is rendered in low definition, only the object in the second ROI region in the historical rendering image can be projected into the first image to obtain a projection frame.

[0186] Optionally, after obtaining the projection frame, the projection frame can also be corrected by a correction network on the up-sampled image, to obtain a corrected projection frame. The correction network is a neural network for filtering an input image, which can filter in the time or spatial dimension, so that the pixel value of the object in the projection frame is more accurate. It can be understood that after the object in the historical rendering image is projected to the first image, the object in the historical rendering image and the object in the first image may overlap, or the object should be masked by other objects, etc. At this time, the pixel value of the object in the projection frame can be corrected by the correction network to obtain the corrected projection frame, so that the pixel value of the object in the projection frame is more reasonable.

[0187] It should be noted that the application does not limit the execution order of steps 304 and 305. Step 304 can be executed first, step 305 can be executed first, or steps 304 and 305 can be executed simultaneously. The specific execution order can be adjusted according to the actual application scenario, which will not be described here.

[0188] 306, fuse the up-sampled image and the projection frame to obtain a target rendering image.

[0189] After obtaining the up-sampled image and the projection frame, the high-definition information in the projection frame is fused into the up-sampled image to obtain a high-definition target rendering image.

[0190] In a possible implementation, the first weight corresponding to each pixel point in the first image and the second weight corresponding to each pixel point in the projection frame can be obtained through a weight coefficient network. The weight coefficient network can be a neural network for calculating the weights corresponding to at least two input images; then, the up-sampled image and the projection frame are fused based on the first weight and the second weight to obtain a target rendering image.

[0191] In a possible implementation, since the first weight is calculated based on the first image, the resolution of the first image is low, and therefore the first weight is interpolated after being obtained to obtain a higher-resolution weight matrix, i.e., a third weight, so that the up-sampled image and the projection frame can be fused using the second weight and the third weight to obtain a higher-resolution target rendering image.

[0192] In a possible implementation, the up-sampled image and the projection frame can also be used as inputs of the weight coefficient network, so that the first weight can be calculated using a higher-resolution up-sampled image, i.e., the weight does not need to be interpolated, and the obtained first weight is more accurate.

[0193] The weight coefficient network can be obtained by training a large number of samples. For example, the samples can include a large number of image pairs and labeled (such as manually labeled or calculated by other means) weight values, so that the weight coefficient network can output more accurate weight values. Compared with calculating the weight values by a large number of calculations, the present application outputs the weight values by the neural network, which can be calculated by the NPU of the device or the shading load of the GPU of the device.

[0194] In a possible implementation, if the first ROI region in the current view region is only low-definition rendered, and the second ROI region in the historical rendering image is projected into the first image to obtain a projection frame, the background part in the current view region can also be high-definition rendered to obtain a high-definition background rendering image, and then the target rendering image fused with the up-sampled image and the projection frame image is fused with the background rendering image, so as to supplement the background part in the target rendering image and obtain an updated target rendering image. Therefore, in the implementation of the present application, the ROI region can be low-definition rendered, the high-definition ROI in the historical rendering image can be reused to improve the definition of the ROI region in the target rendering image, and the calculation resources required for high-definition rendering of the background part are relatively small, so that a high-definition rendering image can be obtained by consuming relatively small calculation resources. Therefore, the method provided in the present application can be deployed on various devices and has strong generalization ability. For example, the method can be adapted to devices with low computing power, such as mobile phones, tablets and other electronic devices, thereby improving the experience of users using these electronic devices.

[0195] 307、high-definition render the first scene data to obtain a second image.

[0196] If it is determined that the current view region is not low-definition rendered, it is determined that the rendering mode of the current view region is high-definition rendering, and the current view region can be high-definition rendered to obtain a second image.

[0197] When the next view region is rendered, the second image can be used as a new historical rendering image to assist in processing subsequent view regions and improving the definition of the image.

[0198] Therefore, in the implementation of the present application, the interval high-definition rendering and low-definition rendering can be used to reduce the computing power requirement, so that the method provided in the present application can be deployed on devices with lower computing power, such as terminal devices, and has strong generalization ability. Therefore, the terminal device can also render a three-dimensional model, such as being applied in a three-dimensional game, an AR application and other scenarios, thereby improving the experience of users of the terminal device. After low-definition rendering, the high-definition information in the projection frame can be reused to improve the definition of the image corresponding to the current view region, thereby obtaining a high-definition rendering image.

[0199] The foregoing describes the flow of the method for rendering a graph provided by the present application. For ease of understanding, the flow of the method for rendering a graph provided by the present application is described in more detail below.

[0200] It should be understood that the method for rendering a graph provided by the present application can be applied in rendering a three-dimensional scene, and the area to be rendered in the three-dimensional scene can change with the movement of a virtual camera. Generally, to improve the user experience, the field of view of the virtual camera can be rendered, and the virtual camera can move, so that the continuously changing field of view of the virtual camera needs to be continuously rendered. In the present application, the field of view in a certain period of time is regarded as a sub-scene to be rendered in a frame, and the current field of view is referred to as the current frame, so that multiple frames of scenes can be continuously rendered. Figure 4 The complete flow of rendering one frame is shown in FIG. 4.

[0201] First, the current field of view of the virtual camera is determined 401.

[0202] The current field of view is the area to be rendered in the three-dimensional scene. For example, when a user uses AR glasses to play an AR game, the AR glasses can be regarded as a virtual camera, and the user can view various areas in the three-dimensional scene by controlling the orientation of the AR glasses. For example, as shown in FIG. 4A, the three-dimensional scene can be understood as a scene surrounding the virtual camera, and the virtual camera can be understood as processing the three-dimensional scene. The field of view angle of the virtual camera can be set according to the actual application scene, and a larger field of view angle can be set in advance, and correspondingly, the range of the current field of view is larger. A smaller field of view angle can also be set in advance, and correspondingly, the range of the current field of view is smaller. Figure 5

[0203] It should be understood that after the current field of view of the virtual camera in the three-dimensional scene is determined, the data corresponding to the current field of view can be extracted from the data corresponding to the three-dimensional scene, i.e., the first scene data, for subsequent rendering.

[0204] Then, the rendering method is determined 402.

[0205] The rendering method can include low-definition rendering 403 and / or high-definition rendering 408. High-definition rendering refers to setting more points for the scene to render, and low-definition rendering refers to setting fewer points for the scene to render. The resolution of the image output by high-definition rendering is higher than that of the image output by low-definition rendering.

[0206] ​Specifically, the rendering manner can be determined in various ways. For example, if a scene switch, or a background switch, occurs between the current view area and the adjacent view area, such as when a user enters a copy, or a new scene, from a current scene while playing a three-dimensional game; or if a motion vector of an object in the current view area is higher than a preset motion value, the motion vector including an offset amount of the object between the adjacent view area and the current view area of the three-dimensional scene, such as an optical flow or a motion speed of the object; or if a change in illumination between the current view area and the adjacent view area exceeds a preset change amount, the information of the illumination including at least one of an illumination intensity, a light source type, a light source incident direction, or a number of light sources, generally, a change in illumination can cause a large change in the three-dimensional model, and thus high-definition rendering is needed to obtain a clearer image; or if an interval between the current view area and a region where high-definition rendering was performed last time is N frames, N being a positive integer.

[0207] Therefore, in the embodiments of the present application, the rendering manner can be adaptively determined according to various scenes. In some scenes with small changes, low-definition rendering can be used, and high-definition information in a rendered image of a previous frame can be reused to obtain a high-definition image of a current frame, so that a high-definition image can be obtained on the basis of reducing the amount of calculation.

[0208] In the present application, high-definition rendering and low-definition rendering of the current view area have different processing procedures. If the rendering manner of the current view area includes low-definition rendering, steps 403-407 are performed as shown in FIG. 4, and if the rendering manner of the current view area is high-definition rendering, step 408 is performed, which will be described below. Figure 4

[0209] I. Low-definition rendering

[0210] 403. Low-definition rendering

[0211] Specifically, the rendering manner can include various manners, such as using a forward rendering pipeline, a deferred rendering pipeline, or AI shading to perform rendering.

[0212] Specifically, the rendering manner can include various manners, such as using a forward rendering pipeline, a deferred rendering pipeline, or AI shading to perform rendering.

[0213] Exemplarily, some possible rendering manners are described below.

[0214] ​For example, the GPU hardware can support a programmable rendering pipeline, which can generally include a vertex shader, a shape (primitive) assembly, a geometry shader, a rasterization or a fragment shader, etc. A commonly used vertex shader is generally used for coordinate transformation of model vertices, a geometry shader can be used for shading of graphics formed by individual vertices, and after rasterization of the geometry graphics output by the geometry shader, a fragment shader is used to calculate the color of the final display pixel and to shade the graphics. Then the shaded graphics output by the fragment shader can be blended to obtain a complete rendering image, and the rendering image can be tested, such as smoothing pixels or denoising, to obtain the final output rendering image.

[0215] For another example, rendering can be performed based on a physical shading model (PBR), and in the fragment shading stage, the shading equation of each pixel is as follows:

[0216]

[0217] where L o is the color of the outgoing light along the direction, L e is the self-luminous color of the object surface along the direction, L i is the color of the object surface along the incident light direction, F is the reflection distribution function of the object surface according to the incident light and the outgoing light, such as where is the normal distribution function, is the geometric function, is the Fresnel function. As can be seen from the calculation formula of the above shading equation, the calculation complexity of the reflection part color of each pixel is linearly related to the number of light sources, and the calculation function is also relatively complex, so the ROI region in the current view angle region can be determined according to the number of light sources of the model, and the region with relatively high calculation complexity in the current view angle region can be accurately found.

[0218] For another example, in the AI field, relatively efficient multiplication and addition operations can be calculated, such as the calculation formula of the convolution kernel as follows:

[0219]

[0220] Using the AI method to perform rendering can reduce the complexity of shading calculation, thereby reducing the load of the GPU.

[0221] For example, in a graphics rendering pipeline, the model can be rendered to different high-definition or low-definition maps respectively by offscreen rendering, that is, the GPU renders a new buffer outside the current screen buffer during rendering, without displaying on the display screen, thereby obtaining the rendered image.

[0222] 404, up-sampling

[0223] Since the current view area is rendered in low definition, the first image obtained has a low resolution, and the first image can be up-sampled to increase the resolution of the image.

[0224] Specifically, up-sampling can be performed by interpolation, such as bilinear interpolation, trilinear interpolation, or other interpolation algorithms such as Bicubic interpolation, or 0 padding interpolation, transposed convolution, or Bicubic interpolation.

[0225] In order to reduce the GPU load, interval low-definition resolution rendering is adopted in the embodiments of the present application. After low-definition rendering, in order to improve the resolution of the final obtained image, the image output by low-definition rendering can be up-sampled to improve the resolution of the image, thereby obtaining an image with higher resolution.

[0226] 405, projection

[0227] The historical rendering image is a high-definition image obtained by rendering the previous frame, and the manner of obtaining the historical rendering image is similar to that of the target rendering image. All or part of the objects in the historical rendering image can be projected into the first image obtained by low-definition rendering, thereby obtaining a high-definition projection frame.

[0228] It can be understood that it is equivalent to projecting the high-definition objects included in the historical rendering image into the first image of the current frame to obtain a projection frame including high-definition information.

[0229] Specifically, the historical rendering image can be projected according to the motion vector, such as calculating the position of the object in the historical rendering image in the first image according to the motion vector, projecting the high-definition object in the historical rendering image into the first image according to the position, and obtaining a high-definition projection frame. It is equivalent to reusing the high-definition information included in the historical rendering image, projecting the high-definition object included in the historical rendering image into the first image, so that the projection frame obtained after projection has high-definition information, which is equivalent to increasing the clarity of each object in the first image.

[0230] For example, Figure 6As shown, each pixel point of the projection frame (t-1) is assigned a motion vector (mv) generated by the renderer, the coordinate position of the object common in the historical rendering image and the current perspective region in the current frame space is calculated, and then a color value is assigned based on the coordinate position to obtain the projected projection frame, that is, the information in the historical rendering image is assigned to the rendering image of the current perspective region, thereby improving the clarity of the rendering image of the current perspective region.

[0231] 406. correction

[0232] The projection frame obtained after projection may have problems such as occlusion or ghosting of part of the object, so the projection frame can usually be corrected to obtain a corrected projection frame.

[0233] Specifically, due to part of the pixel points of the previous frame, the current perspective region can be occluded, resulting in negative problems such as ghosting in the projected image, and the reverse projection projection frame usually needs to be corrected.

[0234] For ease of understanding, an example is introduced. For example, as shown in Figure 7 If the current perspective region includes objects A and B, A and B are independent objects in the historical rendering image, and A and B can partially overlap in the current perspective region. If only A and B in the historical rendering image are projected into the current frame, A and B can appear in the same region, resulting in ghosting of one of the objects, so the projection frame needs to be corrected, such as determining the depth of A and B relative to the virtual camera, then determining which object in the current perspective region is closer to the virtual camera according to the depth, and correcting the pixel value of the overlapping region to the pixel value of the object.

[0235] The specific correction method can include various methods, such as AABB clipping and clamping, convex hull clipping, or variance clipping, etc. A neural network can also be used for correction to reduce the workload of the GPU.

[0236] For example, as shown in Figure 8As shown, taking the AABB clipping as an example, the maximum value and the minimum value of the color of the pixel point of the back projection in the current frame around several pixel points (such as 5 or 9) are calculated to obtain the AABB bounding box of the pixel point, the dashed box of the AABB bounding box represents the color space of the pixel points around the pixel point of the back projection to the current frame, and the colors of the three points (different colors, not shown in the figure) on the edges of the matrix represent the colors of the three points around the projection position. If the color of the projection point in the projection frame is outside the AABB bounding box, it means that the color does not conform to the color distribution of the projected point in the current frame, and needs to be corrected. The color vector can be obtained by directly subtracting the color of the projected point, and then the intersection of the color vector and the AABB bounding box is obtained to obtain the color of the projection point. Finally, interpolation is performed to obtain the final corrected color. Other correction methods are similar, and the difference lies in the different ways of calculating the bounding box, which will not be repeated here.

[0237] Optionally, in a possible implementation, a neural network can be used to implement the correction of the projection frame, which is referred to as a correction network below. The input of the correction network can include the projection frame and the up-sampled image. The correction network can be implemented by various networks, such as CNN, DNN, deep convolutional neural network (DCNN), recurrent neural network (RNN), etc., and can also be a neural network constructed by structure search. The structure of the neural network mentioned in this application and below is not limited.

[0238] The neural network mentioned in this application can be a neural network obtained by training using a training set. For example, a large number of sample pairs can be collected, each sample pair including a projection frame and a current frame with low definition (which can be understood as the aforementioned up-sampled image), and a high-definition corrected image. The correction network is supervised trained through a large number of sample pairs, so that the correction network outputs a corrected higher-definition image.

[0239] For example, the correction network can include multiple convolutional layers. During the training process, the L1 loss can be calculated pixel by pixel between the output image and the high-definition image rendered, and the rules of the corrected image can be learned in a data-driven manner. In the inference phase, the trained network model can be used for inference to output the corrected image.

[0240] 407, fusion

[0241] After obtaining the up-sampled image and the projection frame (or replaced with the corrected projection frame), the up-sampled image and the projection frame can be fused to obtain a higher-definition target rendering image.

[0242] The fusion manner of the up-sampled image and the projection frame can include various manners, such as channel splicing or weighted fusion, and the application exemplarily takes the weighted fusion as an example for exemplarily description, and in an actual application scenario, other fusion manners can be replaced, and the application does not limit this.

[0243] First, before fusing the up-sampled image and the projection frame, the fusion weight can be calculated, that is, the value of the weight of the up-sampled image and the projection frame respectively occupied when fusing.

[0244] For example, the weight of the up-sampled image and the projection frame respectively occupied can be calculated in combination with the motion vector, such as a negative correlation between the motion vector of each object and the weight of the projection frame, that is, the greater the motion vector of the object, the smaller the weight of the region of the object in the projection frame, thereby avoiding the object being unclear due to the motion of the object.

[0245] Optionally, in the embodiment of the application, the weight of the up-sampled image and the projection frame respectively occupied can be output by a neural network, which is referred to as a weight coefficient network below. The weight coefficient network can be implemented by various structures, and the training of the weight coefficient network is performed, such as a CNN, a DNN, a DCNN, an RNN or a regression network, and the neural network can also be constructed by structure search, and the application does not limit this.

[0246] Specifically, in the training phase of the weight coefficient network, a large number of samples can be used to train the weight coefficient network, so that the weight coefficient network can output a single-channel image (referred to as a fusion weight map below) with the same size as the final output image, and each pixel value represents the fusion coefficient of the pixel point. In the inference phase, one of the first image or the up-sampled image and the projection frame can be taken as the input of the weight coefficient network, and the fusion weight map is output. If the first image and the projection frame are taken as the input of the weight coefficient network, the size of the output fusion weight map can be the same as the first image, and then the fusion weight map can be interpolated and enlarged, so that the size of the fusion weight map matches the size of the final output image, and the fusion of the up-sampled image and the projection frame is realized.

[0247] Optionally, the image input to the weight coefficient network can also be reduced in size, so that the image input to the weight coefficient network is smaller, and the calculation amount of the weight coefficient network is reduced. After obtaining the fusion weight map, the fusion weight map can be interpolated and enlarged, so that the size of the fusion weight map matches the size of the final output image, and the fusion of the up-sampled image and the projection frame is realized. Therefore, in this embodiment, the size of the input image of the weight coefficient network can be reduced to reduce the calculation amount of the weight coefficient network, and the resolution of the fusion weight map can be improved by interpolation, so that the size of the final fusion weight map matches the size of the up-sampled image and the projection frame, facilitating subsequent fusion.

[0248] For example, as shown in FIG. 8, the up-sampled image and the corrected projection frame are reduced in size, for example, down-sampled to obtain an image with a smaller resolution, and then the image with a smaller resolution is input to the weight coefficient network, and a fusion weight map corresponding to the image with a smaller resolution is output, and then the fusion weight map is processed, for example, interpolated, to obtain a fusion weight map with a higher resolution, which includes the weight values corresponding to the up-sampled image or the corrected projection frame. Figure 9

[0249] Secondly, after the weights corresponding to the up-sampled image and the projection frame are calculated, the up-sampled image and the projection frame can be fused to obtain a high-definition target rendering image. For example, the pixel value of each pixel point can be represented as: color = a * current + (1-a) * previous, where current represents the value of the pixel point in the up-sampled image, previous represents the value of the corresponding pixel point in the projection frame, and a represents the weight corresponding to the up-sampled image, and (1-a) represents the weight corresponding to the projection frame.

[0250] Some possible neural networks are exemplarily introduced below.

[0251] Generally, the more focused the function of the neural network is, and the more prior knowledge is used, the more the complexity of solving the neural network can be reduced, so the present application fully utilizes prior knowledge to design a fine special AI network, and learns the weights of the neural network in a data-driven manner, instead of using traditional rectification algorithms, interpolation algorithms or manually designed fusion coefficient methods. The fine special AI network greatly reduces the solution space of the network and reduces the complexity of the network. For example, the neural network used can be a lightweight U-Net network for mobile terminals, the input of the rectification network can include a projection frame, and the output can include a corrected projection frame, the input of the weight coefficient network can include an up-sampled image and a corrected projection frame, and the output can be a single-channel fusion weight map, each pixel point of which represents the weight of the corresponding pixel point in the up-sampled image or the corrected projection frame.

[0252] ​In the training stage, the network parameters are solved by optimizing the loss function in an end-to-end data-driven manner, and the calculation method of the loss function can be represented as:

[0253]

[0254] where w1, w2, and w3 are different cumulative weight values, which can be preset values or training updated values, gt represents a high-definition reference frame used in the training stage, prev correct is a projection frame after projection correction, color blend represents the final rendered image after fusion. The last term is used for time domain smoothing of multiple frames, and interpolation is performed on different frames within the range of N frames, so that the output result is smoother.

[0255] When the training is completed, in the inference stage, the trained neural network can be deployed in the device alone, so as to output the corresponding value.

[0256] II. High-definition rendering

[0257] 408. High-definition rendering

[0258] If it is determined to perform high-definition rendering on the current view area, a high-definition target rendering image can be obtained. The rendering process of high-definition rendering can refer to the description of step 403, and the difference is that more points are colored during high-definition rendering than during low-definition rendering, so that a rendering image with higher resolution can be obtained. The difference between high-definition rendering and low-definition rendering is that the number of points rendered is different during rendering. For example, when high-definition rendering and low-definition rendering are performed on the same scene, the number of points of low-definition rendering is less than the number of points of high-definition rendering, so that the resolution of the image obtained by high-definition rendering is higher than the resolution of the image obtained by low-definition rendering, and the image obtained by high-definition rendering is clearer.

[0259] For example, the difference between low-definition rendering and high-definition rendering can be as Figure 10 shown. During low-definition rendering or high-definition rendering, the rendered points are discrete points in time and space. During low-definition rendering, each rendered pixel point is colored once, while during high-definition rendering, each rendered point is colored four times. After rasterization, it is obvious that the resolution of the image obtained by high-definition rendering is higher than the resolution of the image obtained by low-definition rendering, and the image obtained by high-definition rendering includes more information.

[0260] 409. Post-processing

[0261] In addition, after obtaining the high-definition target rendering image by high-definition rendering or low-definition rendering, subsequent processing can be performed on the target rendering image, and the process of post-processing can also be different in different scenarios.

[0262] For example, if the current scene is to render a three-dimensional scene in a VR game, after obtaining the rendered image, the rendered image can be image-optimized or transmitted to a display module for display, etc. Figure 11 as shown.

[0263] For another example, after obtaining the target rendered image, the target rendered image can be saved to a memory, so that the rendered image can be directly obtained from the memory subsequently.

[0264] When the next frame is rendered, the target rendered image can be used as a historical rendered image, so that when the next frame is rendered, the high-definition information included in the historical rendered image can be reused to obtain a high-definition rendered image.

[0265] Therefore, in the embodiments of the present application, high-definition rendering or low-definition rendering can be adaptively selected to render the current frame, so that the amount of calculation required for rendering the three-dimensional scene is reduced by low-definition rendering. Moreover, when low-definition rendering is performed, the high-definition information included in the high-definition historical rendered image can be reused to improve the definition of the image output by low-definition rendering, so that even if low-definition rendering is performed, a high-definition rendered image can be obtained, the amount of calculation required for rendering is reduced, and a high-definition image can also be obtained. Therefore, the method provided by the present application can be adapted to devices with lower computing power, has strong generalization ability, and has high user experience.

[0266] Generally, in some scenes, the foreground part in the current view angle region consumes more calculation amount when being rendered, while the background part can consume less calculation amount when being rendered. Therefore, high-definition rendering or low-definition rendering can be selected for the foreground part, and high-definition rendering can be performed for the background part, and then the foreground part and the background part rendered respectively are fused to obtain high-definition foreground part and high-definition background part. The following will introduce this scene in more detail.

[0267] Referring to Figure 12 , another flowchart of a graphics rendering method provided by the present application is shown.

[0268] The flowchart is similar to the steps in the foregoing Figure 4 , and the similar parts will not be described again. The following will introduce some different steps.

[0269] 1201, determining a current view angle region

[0270] 1202, determining a rendering mode

[0271] 1203, determining an ROI region

[0272] The ROI region (or the first ROI region) can be a region determined based on the complexity of the model in the 3D scene, or it can be a region determined based on user input data, etc., and the model that needs to be rendered in low definition is determined from this ROI region.

[0273] For example, objects in a 3D scene can be pre-classified according to their complexity. Objects that require fewer geometric shapes can be classified as simpler models, while those that require more geometric shapes can be classified as complex models. During rendering, the model complexity of an object can be determined based on its classification, thus determining whether to render it in low resolution.

[0274] For example, the Region of Interest (ROI) can be determined based on information about lighting, shadows, light reflection, or the material of the object in the current view area. Information such as light intensity, light source type, light source incident direction, number of light sources, shadow area, number of shadowed areas, direction of light reflection, or intensity of reflected light can be used to determine the complexity of the model, thus identifying the region containing the most complex model as the ROI.

[0275] For example, the ROI region can be determined based on the area selected by the user. As the method provided in this application can be deployed on a terminal, the user can select an area as the ROI region from the current viewpoint area of ​​the 3D scene via the terminal's touchscreen, thereby enabling rendering according to the user's needs and improving the user experience.

[0276] For example, the ROI region in the current view area can be a high-precision texture region determined according to the user's selection. It can be a workflow texture selected by the user, such as a metal workflow: color map (albedo), metalness map (metalness), roughness map (roughness), normal map (normal); reflectivity-smoothness workflow: diffuse map (diffuse), specular map (specular), glossness (glossness / smoothness), normal map, and other pre-baked effect maps, etc.

[0277] For example, such as Figure 13 As shown, a character with relatively complex rendering is selected as the Region of Interest (ROI) in the 3D scene, and the background of this ROI is filtered out to obtain the character model that needs to be rendered in low resolution. It should be noted that the character model here is only an illustrative example for easy distinction, and in actual application scenarios, this 3D scene may not usually be displayed in the display interface.

[0278] Therefore, in the embodiments of the present application, the region with complex lighting calculation or complex model material in rendering can be taken as the ROI region for interval high-definition rendering or low-definition rendering. Compared with high-definition rendering, the low-definition rendering can significantly reduce the calculation amount required for rendering and reduce the demand for the computing power of the device, so that the device with low computing power can also render the ROI region, and the ROI region of the current view region can be improved through subsequent reuse of the historical rendering image

[0279] 1204, low-definition rendering

[0280] After determining the ROI region in the current view region, the ROI region can be rendered in low definition. The rendering process is similar to the aforementioned step 403, and the only difference is the size of the rendering region, i.e., the ROI region can be smaller than the current view region, which will not be described here.

[0281] 1205, up-sampling

[0282] After rendering the ROI region in the current view region, the image of the rendered ROI region can be up-sampled to obtain an up-sampled image with higher resolution, i.e., a rendered image of the ROI region with higher resolution. The up-sampling process can refer to the aforementioned step 404, which will not be described here.

[0283] 1206, projection

[0284] The difference between the aforementioned step 405 is that the ROI region (or referred to as the second ROI region) in the historical rendering image is projected into the first image to obtain a projection frame. The second ROI region and the aforementioned first ROI region include the same object. For example, if the object in the aforementioned first ROI region includes a cat, the second ROI region also includes the cat.

[0285] The specific projection method is similar to the aforementioned step 405, which will not be described here.

[0286] 1207, correction

[0287] The projection frame obtained after projection may have problems such as occlusion or ghosting of some objects, so the projection frame can usually be corrected to obtain a corrected projection frame. The correction method is similar to the aforementioned step 406, which will not be described here.

[0288] 1208, fusion of ROI region

[0289] After the up-sampling to obtain the rendering image of the ROI region in the current view area with higher resolution, and the rectification of the projection frame to obtain the rectified projection frame, the rendering image of the ROI region in the current view area and the rectified projection frame can be fused, and the fusion process is similar to the foregoing step 407, which will not be described herein again.

[0290] Therefore, in the embodiments of the present application, the ROI region in the current view area can be rendered in low definition, and then the high-definition information of the ROI region included in the historical rendering image can be reused to improve the definition of the rendering image of the ROI region in the current view area, so that even in the case of using low-definition rendering with low computational complexity, the high-definition information in the historical rendering image can be reused to make the rendering image of the ROI region higher in definition, and the high-definition rendering image can be obtained with low computational complexity.

[0291] Exemplarily, as shown in Figure 14 the low-definition rendering image of the ROI region in the current view area is obtained, and the ROI region in the historical rendering image is projected into the current view area to obtain a projection frame. Then, the low-definition rendering image and the projection frame are fused to obtain a clear output image. Obviously, the image obtained by low-definition rendering has lower resolution, and of course, the required computing power is also lower. By fusing the high-resolution projection frame, more details are supplemented to the image obtained by low-definition rendering, and a higher-definition output image is obtained.

[0292] 1209, high-definition rendering

[0293] In addition to the low-definition rendering of the ROI region in the current view area, the background part other than the ROI region in the current view area can also be rendered in high definition to obtain a high-definition background rendering image.

[0294] The way of high-definition rendering of the background part can refer to the foregoing step 408, which will not be described herein again.

[0295] 1210, fusion of rendering images of ROI region and background part

[0296] After obtaining the high-definition rendering images of the ROI region and the background part, the rendering images of the ROI region and the background part can be fused to obtain a complete rendering image of the current view area. Therefore, in the embodiments of the present application, the selected model can be rendered onto a low-definition map, a high-definition map can be obtained by up-sampling and fusing the information included in the historical rendering image, and a complete high-definition rendering image can be obtained by fusing the high-definition map of the ROI region and the high-definition rendering image of the background part.

[0297] The fusion method can include stitching together the rendered images of the ROI region and the background. For example, the rendered image of the ROI region can be obtained by rendering in low resolution and then reusing the high resolution model from the historical rendered image, while the background rendered image can be obtained by rendering the background part of the current view area excluding the ROI region in high resolution. The rendered images of the ROI region and the background rendered image are then stitched together, or the rendered image of the character model is projected onto the background rendered image to obtain a complete high resolution rendered image of the current view area.

[0298] For example, a method for rendering the ROI region in the current view area in low resolution and reusing high resolution information from historically rendered images can be found in [reference needed]. Figure 15 Among them, f t-1 Indicates historical rendered images, f t-1 t This represents the projected frame obtained by projecting the ROI extracted from the historical rendered image onto the current frame, and then correcting the projected frame through a correction network to obtain the corrected projected frame f. t-1 recitify Then, a Region of Interest (ROI) is selected from the current view area for rendering, resulting in a rendered ROI. This rendered ROI is then upsampled, specifically through an upsampling neural network, to reduce GPU load and obtain an upsampled rendered image f. t up Then f t up and f t-1 recitify As input to the regression network (i.e., the weighted coefficient network), the output f t up The corresponding weights α are then fused with f. t up and f t-1 recitify The fusion method can be represented as: f t-out That is, the rendered image of the merged ROI region. Then, the rendered image of the ROI region in the current frame, i.e., f t-out The rendered image is then blended with the background image of the current frame to obtain a complete rendered image.

[0299] 1211, High-definition rendering

[0300] 1212. Post-processing

[0301] Steps 1211 and 1212 can be referred to in steps 408 and 409 above, and will not be repeated here.

[0302] Therefore, in the embodiments of the present application, the calculation amount can be reduced by rendering in high definition and low definition alternately, and the definition of the image obtained by low definition rendering in the current view area can be improved by reusing the high definition information included in the historical rendering image, so that a high definition rendering image can also be obtained with less calculation amount. Therefore, the method provided by the present application can be applied not only to devices with high computing power, but also to devices with low computing power, and has strong generalization ability and can adapt to more hardware devices.

[0303] In addition, in the embodiments of the present application, the prior information of TAA can be fully utilized by using a refined neural network, and a special network is used instead of the traditional GPU calculation method, such as interpolation algorithm, projection frame heuristic correction algorithm or manual weight method. Projection frame correction or fusion weight calculation can be realized by using a neural network respectively, so that the calculation of projection frame correction or fusion weight can be realized by CPU or NPU, so as to reduce the load of GPU and reduce the computing power required by GPU during rendering, so that a terminal device with low computing power can also output a high definition rendering image.

[0304] To facilitate further understanding, the effects of the image rendering method provided by the present application are exemplarily introduced as follows.

[0305] Firstly, when using a platform to produce a game, the ROI area is rendered in 540p, the background area is rendered in 1080p, then the ROI area is up-sampled, and the ROI area in the high definition historical rendering image is fused to obtain a high definition ROI area rendering image, then the ROI area rendering image and the background area rendering image are fused to obtain the final output image. Figure 16 As shown in FIG. 8, a comparison between the rendering image obtained by the method provided by the present application (such as the interval high-low definition rendering shown in FIG. 7) and the rendering image obtained by high definition rendering (such as the high definition rendering shown in FIG. 6). Figure 16 As shown in FIG. 8, a comparison between the rendering image obtained by the method provided by the present application (such as the interval high-low definition rendering shown in FIG. 7) and the rendering image obtained by high definition rendering (such as the high definition rendering shown in FIG. 6). Figure 16 As shown in FIG. 8, a comparison between the rendering image obtained by the method provided by the present application (such as the interval high-low definition rendering shown in FIG. 7) and the rendering image obtained by high definition rendering (such as the high definition rendering shown in FIG. 6).

[0306] In more detail, the rendering effects of the ROI area in the scheme provided by the present application and the commonly used scheme can be as shown in FIG. 9. Figure 17 As shown in FIG. 9, (a) is a rendering image obtained by direct rendering, (b) is an image obtained by TAA rendering, and (c) is a rendering image obtained by the method provided by the present application. Obviously, the rendering image obtained by the method provided by the present application is clearer and has more details.

[0307] The foregoing describes in detail the flow of the graphic rendering method provided in the present application. The following describes the device applying the graphic rendering method provided in the present application in combination with the structure of the terminal.

[0308] First, the structure of the GPU and the NPU in the commonly used terminal can be as shown in Figure 18 , wherein the GPU and the NPU do not share the buffer, and the rendering, up-sampling or fusion and the like are all performed by the GPU, so that the GPU has a large load and cannot perform the graphic rendering in the mobile terminal first.

[0309] The graphic method provided in the present application can be applied to the terminal as shown in Figure 19 , which can include the GPU and the CPU. Of course, in addition to the GPU and the NPU, the terminal can also include other devices such as the display screen or the camera and the like, which are not described herein. Among them, the GPU and the NPU can share the buffer, such as the G-Buffer buffer shown in Figure 19 .

[0310] Specifically, the GPU is configured to render the first scene data to obtain a first image, the first scene data is obtained according to the perspective region of the virtual camera, and the rendering manner includes at least one of a first rendering manner or a second rendering manner, and the resolution of the image obtained by the first rendering manner is higher than the resolution of the image obtained by the second rendering manner.

[0311] The NPU is configured to, when the rendering manner of the first scene data includes the second rendering manner, up-sample the first image to obtain an up-sampled image.

[0312] The GPU is further configured to obtain a historical rendering image, the historical rendering image is obtained by rendering second scene data, the target object exists in both the historical rendering image and the first image, and the resolution of the target object in the historical rendering image is higher than the resolution of the target object in the first image.

[0313] The GPU is further configured to project the target object in the historical rendering image into the first image to obtain a projection frame.

[0314] The NPU is further configured to fuse the up-sampled image and the projection frame to obtain a target rendering image.

[0315] The effects of the second aspect and any optional implementation manner of the present application can be referred to the effects of the foregoing first aspect and any optional implementation manner, which are not described herein.

[0316] In a possible implementation, the GPU is further configured to render data corresponding to a first ROI region of interest in the first scene data using a second rendering manner to obtain a first image, the first ROI region being a preset region or a region determined from a view region corresponding to the first scene data according to a preset manner.

[0317] In a possible implementation, the GPU is further configured to: take a region including the target object in the historical rendering image as a second ROI region; and project a region corresponding to the target object in the second ROI region in the historical rendering image to the first image to obtain a projection frame.

[0318] The NPU is specifically configured to fuse the second ROI region in the projection frame and the first image to obtain a target rendering image.

[0319] In a possible implementation, the GPU is further configured to: render data of a background region in the first scene data by using a first rendering manner to obtain a background rendering image, the background region being a region in the view region corresponding to the first scene data except the first ROI region; and fuse the target rendering image and the background rendering image to obtain an updated target rendering image.

[0320] In a possible implementation, if the current view region meets a preset condition, the GPU is further configured to determine that the rendering manner of the current view region includes the first rendering manner, and the preset condition includes one or more of the following:

[0321] a background switch between the first scene data and an image obtained by rendering last time; or a motion vector of at least one object in the first scene data is higher than a preset motion value, the motion vector including an offset of the at least one object in the first scene data and the second scene data; or a change of illumination between the first scene data and the second scene data exceeds a preset change amount, the information of the illumination including at least one of an illumination intensity, a light source type, a light source incident direction, or a light source number; or a region rendered by using the first rendering manner last time and the second scene data are separated by N frames, N being a positive integer.

[0322] In a possible implementation, the first ROI region is determined based on one or more of information of illumination, information of shadow, information of illumination reflection, or material of the target object of the first scene data.

[0323] In a possible implementation, the NPU is further configured to: obtain, by a weight coefficient network, the first weight corresponding to each pixel point in the first image and the second weight corresponding to each pixel point in the projection frame, the weight coefficient network being a neural network for calculating weights corresponding to inputted at least two frames of images; and fuse the up-sampled image and the projection frame based on the first weight and the second weight to obtain the target rendering image.

[0324] In a possible implementation, the NPU is further configured to, before obtaining the target rendering image by fusing the up-sampled image and the projection frame, correct the projection frame by a correction network to obtain a corrected projection frame, the correction network being a neural network for filtering inputted images.

[0325] In a possible implementation, the GPU is further configured to, before rendering the first scene data to obtain the first image, render the second scene data by a first rendering mode to obtain a historical rendering image.

[0326] For example, the rendering steps in the method provided in the present application can be performed by the GPU, such as the steps 301-303, 305, 307 shown in the foregoing Figure 3 steps 403, 405, 408, or 409 shown in the foregoing Figure 4 steps 1203, 1204, 1206, 1209, or 1211 shown in the foregoing Figure 12 steps. In addition to the rendering steps, the steps can be performed by the NPU, such as the steps 304, 306 shown in the foregoing Figure 3 steps 404, 406, 407 shown in the foregoing Figure 4 steps 1207, 1208, or 1210 shown in the foregoing Figure 12 steps, which can be implemented by a neural network and can be performed by the NPU, thereby reducing the load of the GPU. The GPU and the NPU can exchange data through a G-Buffer, such as the GPU can store the image obtained after low-resolution rendering or high-definition rendering or the image obtained after projection in the G-Buffer, and the NPU can read the image obtained after low-resolution rendering or high-definition rendering or the image obtained after projection stored by the GPU from the G-Buffer to perform subsequent processing, such as correction, fusion, and the like.

[0327] Therefore, in the terminal provided in the present application, the NPU can be used to perform part of the calculation in the rendering process, such as rectification of the projection frame or image fusion, so as to reduce the demand for the computing power of the GPU when rendering, and on the basis of the low required computing power, the high-definition information in the historical rendering image can be reused to obtain the high-definition rendering image of the current view area. Therefore, the method for rendering graphics provided in the present application can be deployed not only in devices with high computing power, but also in terminals with low computing power, thereby improving the experience of users of the terminal with low computing power.

[0328] With reference to Figure 20 The present application also provides a device for rendering graphics, comprising:

[0329] The rendering module 2002 is configured to render the first scene data to obtain a first image, the first scene data being obtained according to a view area of a virtual camera, and the rendering mode comprising at least one of a first rendering mode or a second rendering mode, the resolution of the image obtained by the first rendering mode being higher than the resolution of the image obtained by the second rendering mode.

[0330] The upsampling module 2003 is configured to, if the rendering mode of the current view area comprises the second rendering mode, perform upsampling on the first image to obtain an upsampled image.

[0331] The obtaining module 2001 is configured to obtain a historical rendering image, the historical rendering image being obtained by rendering second scene data, the target object being present in both the historical rendering image and the first image, and the resolution of the target object in the historical rendering image being higher than the resolution of the target object in the first image.

[0332] The projection module 2004 is configured to project the target object in the historical rendering image into the first image to obtain a projection frame.

[0333] In a possible implementation, the rendering module 2002 is specifically configured to use the second rendering mode to render data corresponding to a first ROI region of interest in the first scene data to obtain the first image, the first ROI region being a preset region or a region determined from the view area corresponding to the first scene data according to a preset manner.

[0334] In a possible implementation, the projection module 2004 is specifically configured to take a region including the target object in the historical rendering image as a second ROI region, and project the region corresponding to the target object in the second ROI region in the historical rendering image into the first image to obtain the projection frame.

[0335] The fusion module 2005 is further configured to fuse the second ROI region in the projection frame and the first image to obtain a target rendering image, the second ROI region and the first ROI region including the same object.

[0336] In a possible implementation, the rendering module 2002 is specifically configured to render data of a background region in the first scene data by using the first rendering manner, to obtain a background rendering image, the background region being a region in a perspective region corresponding to the first scene data except the first ROI region.

[0337] The fusion module 2005 is further configured to fuse the target rendering image and the background rendering image, to obtain an updated target rendering image.

[0338] In a possible implementation, the apparatus can further include a determination module 2006 configured to determine that the rendering manner of the first scene data includes the first rendering manner if the first scene data meets a preset condition, the preset condition including one or more of the following: background switching between the first scene data and an image obtained by rendering last time; or a motion vector of at least one object in the first scene data being higher than a preset motion value, the motion vector including an offset of the at least one object in the first scene data and the second scene data; or a change in illumination between the first scene data and the second scene data exceeding a preset change amount, the information of the illumination including at least one of illumination intensity, light source type, light source incident direction, or light source number; or an interval of N frames between the second scene data and a region rendered using the first rendering manner last time, N being a positive integer.

[0339] In a possible implementation, the first ROI region is determined based on one or more of information of illumination, information of shadow, information of illumination reflection, or material of the target object of the first scene data.

[0340] In a possible implementation, the fusion module 2005 is specifically configured to obtain, by using a weight coefficient network, a first weight corresponding to each pixel point in the first image and a second weight corresponding to each pixel point in the projection frame, the weight coefficient network being a neural network configured to calculate weights corresponding to at least two input images; and fuse the up-sampling image and the projection frame based on the first weight and the second weight, to obtain the target rendering image.

[0341] In a possible implementation, the apparatus further includes a correction module 2007 configured to correct the projection frame by using a correction network before fusing the up-sampling image and the projection frame to obtain the target rendering image, the correction network being a neural network configured to filter an input image.

[0342] In a possible implementation, the rendering module 2002 is further configured to render the second scene data by using the first rendering manner, to obtain a historical rendering image, before rendering the first scene data to obtain the first image.

[0343] Referring to Figure 21 , another structural schematic diagram of a graphics rendering device is provided as follows.

[0344] The training device can include a processor 2101 and a memory 2102. The processor 2101 and the memory 2102 are interconnected by a line. Among them, the memory 2102 stores program instructions and data.

[0345] The memory 2102 stores the program instructions and data corresponding to the steps in the foregoing Figure 7 - Figure 14 . More specifically, the processor can also be a processor for processing images, such as a GPU or a CPU for processing images, etc.

[0346] The processor 2101 is configured to execute the method steps performed by the graphics rendering device shown in any of the embodiments of the foregoing Figure 7 - Figure 14 .

[0347] Optionally, the graphics rendering device can also include a transceiver 2103 for receiving or sending data.

[0348] The computer readable storage medium provided in the embodiments of the present application stores programs, and when the programs are run on a computer, the computer executes the steps in the method described in the embodiments of the foregoing Figure 7 - Figure 14 .

[0349] Optionally, the graphics rendering device shown in the foregoing Figure 21 is a chip.

[0350] The embodiments of the present application also provide a graphics rendering device, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is configured to execute the method steps shown in any of the embodiments of the foregoing Figure 3 - Figure 17 .

[0351] The embodiments of the present application also provide a digital processing chip. The digital processing chip integrates a circuit for realizing the functions of the processor 2101, the processor 2201, or the processor 2101, the processor 2201, and one or more interfaces. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, the digital processing chip can be connected with an external memory through a communication interface. The digital processing chip realizes the method steps in the foregoing embodiments according to the program codes stored in the external memory.

[0352] The embodiment of the present application further provides a computer program product comprising the computer program product, when running on a computer, causes the computer to execute the steps of the method described in any of the preceding Figure 3 - Figure 17

[0353] The image processing device or the training device provided by the embodiment of the present application can be a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the server executes the image processing method described in any of the preceding Figure 3 - Figure 17 The storage unit can be a storage unit in the chip, for example, a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, for example, a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0354] Specifically, the aforementioned processing unit or processor can include a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0355] In addition, it should be noted that the device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. ​

[0356] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software necessary general hardware, of course, can also be realized by special hardware including special integrated circuit, special CPU, special memory, special component and the like. Generally speaking, the functions completed by the computer program can be easily realized by the corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuit, digital circuit or special circuit and the like. However, for the present application, the software program implementation is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product in essence or the part of the prior art. The computer software product is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in various embodiments of the present application.

[0357] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of computer program product in whole or in part.

[0358] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be stored by the computer or the data storage device such as server, data center and the like integrated with one or more available media. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) and the like.

[0359] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a generalised use of these terms to refer to similar elements independently of any specific sequential or chronological order. It is to be understood that the data used herein is to be taken in a generic and descriptive sense and not necessarily in a limiting sense. Furthermore, the terms "comprising", "having", "including", and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to", "comprising, but not limited to", "having, but not limited to", or "including, but not limited to") unless otherwise noted. It is to be understood that such terms are merely to be construed as specifying the presence of what follows the term, but does not preclude the presence of additional, non-recited items.

[0360] Finally, it should be noted that the above-mentioned merely preferred embodiments of the present application are illustrative, and not restrictive, rather, the scope of the present application should be given by the appended claims along with full scope of equivalents to which such claims are entitled.

Claims

1. A graphics rendering method, characterized in that, include: The first scene data is rendered to obtain a first image. The first scene data is obtained based on the viewpoint area of ​​the virtual camera. The rendering method includes at least one of a first rendering method or a second rendering method. The resolution of the image obtained by the first rendering method is higher than the resolution of the image obtained by the second rendering method. If the rendering method of the first scene data includes the second rendering method, then the first image is upsampled to obtain an upsampled image; Acquire historical rendering images, which are obtained by rendering second scene data. The historical rendering images and the first image share a common target object. The resolution of the target object in the historical rendering images is higher than the resolution of the target object in the first image. The second scene data is the scene data corresponding to the previous view area or the adjacent view area of ​​the current view area of ​​the virtual camera. The target object in the historical rendered image is projected onto the first image to obtain a projection frame; The upsampled image and the projected frame are fused to obtain the target rendered image.

2. The method according to claim 1, characterized in that, The process of rendering the first scene data to obtain the first image includes: The data corresponding to the first ROI in the first scene data is rendered using the second rendering method to obtain the first image. The first ROI is a preset area or an area determined from the view area corresponding to the first scene data according to a preset method.

3. The method according to claim 2, characterized in that, The step of projecting the target object from the historical rendered image onto the first image to obtain a projection frame includes: The region in the historical rendered image that includes the target object is designated as the second ROI; The region corresponding to the target object in the second ROI of the historical rendered image is projected onto the first image to obtain the projection frame; The process of fusing the upsampled image and the projected frame to obtain the target rendered image includes: The target rendered image is obtained by fusing the second ROI and the first image in the projection frame.

4. The method according to claim 3, characterized in that, The method further includes: The background region data in the first scene data is rendered using the first rendering method to obtain a background rendering image. The background region is the region outside the first ROI in the view area corresponding to the first scene data. The target rendered image and the background rendered image are merged to obtain the updated target rendered image.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the first scene data meets preset conditions, the rendering method of the first scene data is determined to include the first rendering method, and the preset conditions include one or more of the following: Background switching between the first scene data and the data corresponding to the previous view area of ​​the virtual camera; Alternatively, the motion vector of at least one object in the first scene data is higher than a preset motion value, and the motion vector includes the offset of the at least one object in the first scene data and the second scene data; Alternatively, the change in illumination between the first scene data and the second scene data exceeds a preset amount of change, and the illumination information includes at least one of illumination intensity, light source type, light source incident direction, or number of light sources; Alternatively, the second scene data may be spaced N frames apart from the region that was previously rendered using the first rendering method, where N is a positive integer.

6. The method according to any one of claims 2-4, characterized in that, The first ROI is determined based on one or more of the following: lighting information, shadow information, light reflection information, or the material of the target object from the first scene data.

7. The method according to any one of claims 1-4, characterized in that, The process of fusing the upsampled image and the projected frame to obtain the target rendered image includes: The first weight corresponding to each pixel in the first image and the second weight corresponding to each pixel in the projection frame are obtained through a weight coefficient network. The weight coefficient network is a neural network used to calculate the weights corresponding to at least two input frames of images. The target rendered image is obtained by fusing the upsampled image and the projected frame based on the first weight and the second weight.

8. The method according to any one of claims 1-4, characterized in that, Before fusing the upsampled image and the projected frame to obtain the target rendered image, the method further includes: The upsampled image is corrected by the correction network to obtain the corrected projection frame. The correction network is a neural network used to filter the input image.

9. The method according to any one of claims 1-4, characterized in that, Before rendering the first scene data to obtain the first image, the method further includes: The historical rendered image is obtained by rendering the second scene data using the first rendering method.

10. A terminal device, characterized in that, Including GPU and NPU, The GPU is used to render the first scene data to obtain a first image. The first scene data is obtained based on the viewpoint area of ​​the virtual camera. The rendering method includes at least one of a first rendering method or a second rendering method. The resolution of the image obtained by the first rendering method is higher than the resolution of the image obtained by the second rendering method. The NPU is used to upsample the first image to obtain an upsampled image when the rendering method of the first scene data includes the second rendering method. The GPU is also used to acquire historical rendering images, which are obtained by rendering second scene data. The historical rendering images and the first image have a common target object. The resolution of the target object in the historical rendering images is higher than the resolution of the target object in the first image. The second scene data is the scene data corresponding to the previous view area or the adjacent view area of ​​the current view area of ​​the virtual camera. The GPU is also used to project the target object in the historical rendered image onto the first image to obtain a projection frame; The NPU is also used to fuse the upsampled image and the projected frame to obtain the target rendered image.

11. The terminal device according to claim 10, characterized in that, The GPU is further configured to render the data corresponding to the first ROI in the first scene data using the second rendering method to obtain the first image, wherein the first ROI is a preset region or a region determined from the viewpoint region corresponding to the first scene data according to a preset method.

12. The terminal device according to claim 11, characterized in that, The GPU is further configured to use the region in the historical rendered image that includes the target object as a second ROI; and to project the region corresponding to the target object in the second ROI in the historical rendered image onto the first image to obtain the projection frame; The NPU is specifically used to fuse the second ROI and the first image in the projection frame to obtain the target rendering image.

13. The terminal device according to claim 12, characterized in that, The GPU is also used for: The background region data in the first scene data is rendered using the first rendering method to obtain a background rendering image. The background region is the region outside the first ROI in the view area corresponding to the first scene data. The target rendered image and the background rendered image are merged to obtain the updated target rendered image.

14. The terminal device according to any one of claims 10-13, characterized in that, When the current view area meets preset conditions, the GPU is further configured to determine that the rendering method of the current view area includes the first rendering method, and the preset conditions include one or more of the following: Background switching between the first scene data and the data corresponding to the previous view area of ​​the virtual camera; Alternatively, the motion vector of at least one object in the first scene data is higher than a preset motion value, and the motion vector includes the offset of the at least one object in the first scene data and the second scene data; Alternatively, the change in illumination between the first scene data and the second scene data exceeds a preset amount of change, and the illumination information includes at least one of illumination intensity, light source type, light source incident direction, or number of light sources; Alternatively, the second scene data may be spaced N frames apart from the region that was previously rendered using the first rendering method, where N is a positive integer.

15. The terminal device according to any one of claims 11-13, characterized in that, The first ROI is determined based on one or more of the following: lighting information, shadow information, light reflection information, or the material of the target object from the first scene data.

16. The terminal device according to any one of claims 10-13, characterized in that, The NPU is also used for: The first weight corresponding to each pixel in the first image and the second weight corresponding to each pixel in the projection frame are obtained through a weight coefficient network. The weight coefficient network is a neural network used to calculate the weights corresponding to at least two input frames of images. The target rendered image is obtained by fusing the upsampled image and the projected frame based on the first weight and the second weight.

17. The terminal device according to any one of claims 10-13, characterized in that, The NPU is also used to correct the projection frame by means of the upsampled image before fusing the upsampled image and the projection frame to obtain the target rendering image, so as to obtain the corrected projection frame. The correction network is a neural network used to filter the input image.

18. The terminal device according to any one of claims 10-13, characterized in that, The GPU is further configured to render the second scene data using the first rendering method before rendering the first scene data to obtain the first image, thereby obtaining the historical rendered image.

19. A graphics rendering apparatus, characterized in that, include: A rendering module is used to render first scene data to obtain a first image. The first scene data is obtained based on the viewpoint area of ​​a virtual camera. The rendering method includes at least one of a first rendering method or a second rendering method. The resolution of the image obtained by the first rendering method is higher than the resolution of the image obtained by the second rendering method. The upsampling module is used to upsample the first image to obtain an upsampled image when the rendering method of the current view area includes the second rendering method; The acquisition module is used to acquire historical rendered images, which are obtained by rendering second scene data. The historical rendered images and the first image have a common target object. The resolution of the target object in the historical rendered images is higher than the resolution of the target object in the first image. The second scene data is the scene data corresponding to the previous view area or the adjacent view area of ​​the current view area of ​​the virtual camera. The projection module is used to project the target object in the historical rendered image onto the first image to obtain a projection frame; The fusion module is used to fuse the upsampled image and the projected frame to obtain the target rendered image.

20. The apparatus according to claim 19, characterized in that, The rendering module is specifically used to render the data corresponding to the first ROI in the first scene data using the second rendering method to obtain the first image. The first ROI is a preset area or an area determined from the view area corresponding to the first scene data according to a preset method.

21. The apparatus according to claim 20, characterized in that, The projection module is specifically used to take the region containing the target object in the historical rendered image as the second ROI, and project the region corresponding to the target object in the second ROI in the historical rendered image onto the first image to obtain the projection frame; The fusion module is further configured to fuse the second ROI and the first image in the projection frame to obtain the target rendering image, wherein the second ROI and the first ROI include the same object.

22. The apparatus according to claim 21, characterized in that, The rendering module is specifically used to render the background area data in the first scene data using the first rendering method to obtain a background rendering image. The background area is the area outside the first ROI in the view area corresponding to the first scene data. The fusion module is also used to fuse the target rendered image and the background rendered image to obtain an updated target rendered image.

23. The apparatus according to any one of claims 19-22, characterized in that, The device further includes: The determining module is configured to determine, when the first scene data meets preset conditions, the rendering method of the first scene data includes the first rendering method, wherein the preset conditions include one or more of the following: Background switching between the first scene data and the data corresponding to the previous view area of ​​the virtual camera; Alternatively, the motion vector of at least one object in the first scene data is higher than a preset motion value, and the motion vector includes the offset of the at least one object in the first scene data and the second scene data; Alternatively, the change in illumination between the first scene data and the second scene data exceeds a preset amount of change, and the illumination information includes at least one of illumination intensity, light source type, light source incident direction, or number of light sources; Alternatively, the second scene data may be spaced N frames apart from the region that was previously rendered using the first rendering method, where N is a positive integer.

24. The apparatus according to any one of claims 20-22, characterized in that, The first ROI is determined based on one or more of the following: lighting information, shadow information, light reflection information, or the material of the target object from the first scene data.

25. The apparatus according to any one of claims 19-22, characterized in that, The fusion module is specifically used for: The first weight corresponding to each pixel in the first image and the second weight corresponding to each pixel in the projection frame are obtained through a weight coefficient network. The weight coefficient network is a neural network used to calculate the weights corresponding to at least two input frames of images. The target rendered image is obtained by fusing the upsampled image and the projected frame based on the first weight and the second weight.

26. The apparatus according to any one of claims 19-22, characterized in that, The device further includes: A correction module is used to correct the projection frame by means of the upsampled image before fusing the upsampled image and the projection frame to obtain the target rendering image, so as to obtain a corrected projection frame. The correction network is a neural network used to filter the input image.

27. The apparatus according to any one of claims 19-22, characterized in that, The rendering module is further configured to render the second scene data using the first rendering method before rendering the first scene data to obtain the first image, thereby obtaining the historical rendering image.

28. A graphics rendering apparatus, characterized in that, The method includes a processor coupled to a memory storing a program, wherein when the program instructions stored in the memory are executed by the processor, the method steps of any one of claims 1-9 are implemented.

29. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the steps of the method as claimed in any one of claims 1 to 9.

30. A computer program product, characterized in that, The computer program product includes software code for performing the steps of the method as described in any one of claims 1 to 9.

Images