Training, rendering methods, devices, and electronic equipment for rendering networks

Abstract

This disclosure provides training methods, rendering apparatus, and electronic devices for rendering networks, relating to the field of image processing, particularly to artificial intelligence technologies such as metaverse, image rendering, and processing. The specific implementation scheme is as follows: The color value of each first pixel in a first rendered image is obtained. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses, and the first rendered image is a color image. Based on the sample camera poses and a second mesh model, the world coordinates and first ray direction of the second pixel are determined. The first ray direction is the ray direction from the viewpoint under the sample camera poses to the second pixel. The world coordinates and first ray direction of the second pixel are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieving high-quality 3D rendering.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to artificial intelligence technologies such as metaverse and image rendering processing. Specifically, this disclosure relates to a training method, rendering apparatus, and electronic device for a rendering network. Background Technology

[0002] With the development of image rendering technology, 3D rendering has been widely used. 3D rendering refers to rendering a 3D model into a highly realistic 2D image after the 3D model has been created.

[0003] As users demand higher and higher quality rendered images, how to render high-quality images has become an important technical issue. Summary of the Invention

[0004] To address at least one of the aforementioned deficiencies, this disclosure provides a method, apparatus, and electronic device for training and rendering a rendering network.

[0005] According to a first aspect of this disclosure, a method for training a rendering network is provided, the method comprising:

[0006] Obtain the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image.

[0007] Based on the sample camera pose and the second mesh model, the world coordinates and the direction of the first ray of the second pixel are determined. The direction of the first ray is the direction of the first ray pointing from the viewpoint under the sample camera pose to the second pixel. The second pixel is the pixel in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0008] The rendering network is trained by taking the world coordinates of the second pixel and the direction of the first ray as input and the color value of the first pixel as output.

[0009] According to a second aspect of this disclosure, a rendering method is provided, the method comprising:

[0010] Obtain the second mesh model of the target virtual object and the preset current camera pose;

[0011] Based on the current camera pose and the second mesh model, determine the world coordinates of the third pixel and the direction of the second ray. The direction of the second ray is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is the pixel in the third rendered image rendered by the second mesh model under the current camera pose.

[0012] Based on the world coordinates of the third pixel and the direction of the second ray, the color value of the third pixel is determined to determine the target rendering image. The image difference between the target rendering image and the fourth rendering image meets the preset difference condition. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0013] According to a third aspect of this disclosure, a training apparatus for rendering a network is provided, the apparatus comprising:

[0014] The color value acquisition module is used to acquire the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image.

[0015] The world coordinates and ray direction determination module is used to determine the world coordinates and the first ray direction of the second pixel based on the sample camera pose and the second mesh model. The first ray direction is the ray direction from the viewpoint under the sample camera pose to the second pixel. The second pixel is the pixel in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0016] The rendering network training module is used to train the rendering network by taking the world coordinates of the second pixel and the direction of the second ray as input and the color value of the first pixel as output.

[0017] According to a fourth aspect of this disclosure, a rendering apparatus is provided, the apparatus comprising:

[0018] The rendering resource acquisition module is used to acquire the second mesh model of the target virtual object and the preset current camera pose;

[0019] The world coordinates and ray direction determination module is used to determine the world coordinates and second ray direction of the third pixel based on the current camera pose and the second mesh model. The second ray direction is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is the pixel in the third rendered image rendered by the second mesh model under the current camera pose.

[0020] The rendering module is used to determine the color value of the third pixel based on the world coordinates of the third pixel and the direction of the second ray, so as to determine the target rendering image. The image difference between the target rendering image and the fourth rendering image meets the preset difference conditions. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0021] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0022] At least one processor; and

[0023] A memory communicatively connected to at least one of the aforementioned processors; wherein,

[0024] The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the training or rendering method of the rendering network.

[0025] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the training or rendering method of the rendering network described above.

[0026] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the training or rendering method of the rendering network described above.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a flowchart illustrating a training method for a rendering network provided in an embodiment of this disclosure;

[0030] Figure 2This is a flowchart illustrating a rendering method provided in an embodiment of this disclosure;

[0031] Figure 3 This is a schematic diagram of the structure of a training device for a rendering network provided in an embodiment of this disclosure;

[0032] Figure 4 This is a schematic diagram of the structure of a rendering apparatus provided in an embodiment of this disclosure;

[0033] Figure 5 This is a block diagram of an electronic device used to implement the training or rendering method of the rendering network in the embodiments of this disclosure. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] With the development of technologies such as virtual reality and metaverse, users have increasingly higher demands for the quality of rendered images in virtual scenes. How to render high-quality images has become an important technical problem.

[0036] To render high-quality images, high-precision 3D models are generally required. However, high-precision 3D models have a large data volume, which is not convenient for data transmission. Furthermore, high-precision 3D models have high requirements for the performance of the device's GPU, and rendering high-precision 3D models also consumes a lot of computing power.

[0037] Currently, 3D rendering is generally implemented on terminal devices. The GPU performance of terminal devices may be weak and unable to support the rendering of high-precision 3D models. Therefore, how to render high-quality images on terminal devices with weak GPU performance has become an important technical problem.

[0038] The training and rendering methods, apparatuses, and electronic devices for rendering networks provided in this disclosure are intended to solve at least one of the above-mentioned technical problems in the prior art.

[0039] Figure 1 A flowchart illustrating a training method for a rendering network provided in an embodiment of this disclosure is shown, as follows: Figure 1 As shown, the method can mainly include:

[0040] Step S110: Obtain the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image.

[0041] Step S120: Based on the sample camera pose and the second mesh model, determine the world coordinates of the second pixel and the direction of the first ray. The direction of the first ray is the ray direction from the viewpoint under the sample camera pose to the second pixel. The second pixel is the pixel in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0042] Step S130: Train the rendering network by taking the world coordinates of the second pixel and the direction of the first ray as input and the color value of the first pixel as output.

[0043] The target virtual object is the object that needs to be rendered, such as virtual buildings, virtual trees, virtual characters, etc.

[0044] Both the first mesh model and the second mesh model are three-dimensional mesh models constructed for the target virtual object. The first mesh model has more triangles (i.e. more vertices) than the second mesh model, has a more complex structure, and is richer in detail. The first rendered image rendered by the first mesh model is smoother and more detailed.

[0045] As an example, the first mesh model can be a high-precision model, and the second mesh model can be a low-precision model.

[0046] In this embodiment, a large number of sample camera poses can be preset, and the first mesh model can be rendered under each sample camera pose to obtain a first rendered image under multiple sample camera poses. The first pixel is a pixel in the first rendered image. The first rendered image is a color image, so the color value of each first pixel can be obtained.

[0047] As an example, the color value of the first pixel can be in Red Green Blue (RGB) format.

[0048] In this embodiment of the disclosure, the second pixel is a pixel in the second rendered image rendered by the second mesh model under the pose of each sample camera. Since both the first mesh model and the second mesh model are constructed for the target virtual object, the pixels in the first rendered image rendered by the first mesh model and the pixels in the second rendered image rendered by the second mesh model are in one-to-one correspondence. In other words, the second pixel is in one-to-one correspondence with the first pixel.

[0049] In this embodiment of the disclosure, the world coordinates of the second pixel are the coordinates of the second pixel in the world coordinate system, which can be determined based on the sample camera pose and the second mesh model.

[0050] In this embodiment of the present disclosure, the direction of the first ray pointing from the viewpoint to the second pixel can be calculated based on the sample camera pose and the pinhole camera mathematical formula.

[0051] As an example, the direction of the first ray can be represented by a unit vector.

[0052] For any second pixel, the direction of the first ray pointing from the viewpoint to that second pixel is the observation direction for that second pixel. By setting a large number of sample camera poses, a large number of sample camera directions can be obtained, that is, a large number of different observation directions for the second pixel can be obtained.

[0053] During rendering, a pre-defined texture map is typically used to replace the surface of the mesh model. The texture color of a pixel can be obtained from its world coordinates. However, because different materials reflect light differently, the actual color of a pixel will change when viewed from different angles. Therefore, this solution can predict the color value of the rendered pixel by analyzing the color changes caused by different camera poses, thus achieving image prediction.

[0054] In this embodiment of the disclosure, the world coordinates of the second pixel, the sample shooting direction under a certain sample camera pose, and the color value of the first pixel corresponding to the second pixel can be constructed as a sample data. By constructing sample data for the second pixel in each second rendered image, a sample dataset containing multiple sample data is obtained, and then the rendering network is trained based on the sample dataset.

[0055] Specifically, the world coordinates of the second pixel and the sample camera direction can be input into the rendering network to obtain the color prediction value of the second pixel. A loss function is constructed based on the difference between the color prediction value and the color value of the corresponding first pixel. The loss value is backpropagated and the network weight parameters of the rendering network are updated until the loss function converges, and the trained rendering network is obtained.

[0056] As an example, the loss function in this case can be constructed based on the L2 loss function and the structure similarity (SSIM) function.

[0057] As an example, the rendering network can be a neural network, which can take the world coordinates of the second pixel and the unit vector of the sample camera direction as input to the neural network, and take the RGB value of the color value of the first pixel as output to the neural network.

[0058] The method provided in this disclosure obtains the color values ​​of each first pixel in a first rendered image. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses, and the first rendered image is a color image. Based on the sample camera poses and a second mesh model, the world coordinates of a second pixel and the direction of a first ray pointing from the viewpoint under the sample camera poses to the second pixel are determined. The second pixel is a pixel in the second rendered image rendered by the second mesh model under the sample camera poses. Both the first and second mesh models are mesh models of the target virtual object, and the number of triangular faces in the first mesh model is greater than the number of triangular faces in the second mesh model. The world coordinates of the second pixel and the direction of the first ray are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieve high-quality 3D rendering, and help better meet users' needs for high-quality rendered images.

[0059] In this embodiment of the disclosure, by using a rendering network, it is possible to predict high-quality rendered images based on a low-precision second mesh model, thereby reducing the performance requirements of the terminal device's GPU and enabling the rendering of high-quality rendered images on terminal devices with weak GPU performance.

[0060] In this solution, a low-precision second mesh model is used during rendering. The data volume of the second mesh model is not too large, which facilitates data transmission and greatly reduces the time required for terminal devices to download materials. Furthermore, the computing power required to predict high-quality rendered images based on the rendering network in this solution is also significantly reduced compared to directly rendering high-precision 3D models.

[0061] In one optional embodiment of this disclosure, the second rendered image is a depth map. Based on the sample camera pose and the second mesh model, the world coordinates of the second pixel are determined, including:

[0062] The second mesh model is rendered under the sample camera pose to obtain the second rendered image;

[0063] Obtain the depth value of the second pixel from the second rendered image;

[0064] The world coordinates of the second pixel are determined based on the depth value of the second pixel and the pose of the sample camera.

[0065] In this embodiment of the disclosure, the second rendered image is a depth map, that is, a depth map of the second mesh model is rendered under each sample camera pose. Then, the depth map can be traversed to obtain the depth value of each second pixel.

[0066] In this embodiment of the disclosure, the world coordinates of the second pixel can be determined based on the depth value of the second pixel and the sample camera pose. Specifically, a unit vector from the viewpoint in the camera pose to the second pixel can be found. Multiplying the unit vector by the depth value of the second pixel yields the vector from the viewpoint to the second pixel. Adding this vector to the actual position of the camera gives the world coordinates of the second pixel.

[0067] In one optional embodiment of this disclosure, before obtaining the color value of each first pixel in the first rendered image, the method further includes:

[0068] The first mesh model is rendered using preset texture maps and preset shaders under multiple sample camera poses to obtain the first rendered image.

[0069] In this embodiment of the disclosure, a preset texture map and a preset shader can be used to render the first mesh model to obtain a color image (i.e., the first rendered image).

[0070] A shader is a program that runs on the GPU and is used to perform image rendering (calculating lighting, brightness, color, etc.). Multiple shaders can be pre-configured, and the shader used to render the first mesh model can be selected based on actual needs.

[0071] In one alternative embodiment of this disclosure, the second mesh model is obtained by subtracting the surface area from the first mesh model.

[0072] In this embodiment of the disclosure, the second mesh model can be obtained by reducing the surface area of ​​the first mesh model.

[0073] As an example, the face reduction process in this case is implemented using a simplified algorithm based on the Quadric Error Mactrics (QEM) model.

[0074] Figure 2 A flowchart illustrating a rendering method provided in an embodiment of this disclosure is shown, as follows: Figure 2 As shown, the method can mainly include:

[0075] Step S210: Obtain the second mesh model of the target virtual object and the preset current camera pose;

[0076] Step S220: Based on the current camera pose and the second mesh model, determine the world coordinates of the third pixel and the direction of the second ray. The direction of the second ray is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is the pixel in the third rendered image rendered by the second mesh model under the current camera pose.

[0077] Step S230: Based on the world coordinates of the third pixel and the direction of the second ray, determine the color value of the third pixel to determine the target rendering image. The image difference between the target rendering image and the fourth rendering image meets the preset difference condition. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0078] The target virtual object is the object that needs to be rendered, such as virtual buildings, virtual trees, virtual characters, etc.

[0079] The current camera pose is the camera pose used for rendering the target virtual object.

[0080] Both the first mesh model and the second mesh model are three-dimensional mesh models constructed for the target virtual object. The first mesh model has more triangles (i.e. more vertices) than the second mesh model, has a more complex structure, and is richer in detail. The third rendered image rendered by the first mesh model is smoother and more detailed.

[0081] As an example, the first mesh model can be a high-precision model, and the second mesh model can be a low-precision model.

[0082] In this embodiment, the third pixel is a pixel in the third rendered image rendered by the second mesh model under the current camera pose. The world coordinates of the third pixel are the coordinates of the third pixel in the world coordinate system, which can be determined based on the current camera pose and the second mesh model.

[0083] In this embodiment of the present disclosure, the direction of the second ray pointing from the viewpoint in the current camera pose to the third pixel can be calculated based on the sample camera pose and the pinhole camera mathematical formula.

[0084] As an example, the direction of the second ray can be characterized by a unit vector.

[0085] For any third pixel, the direction of the second ray pointing from the viewpoint of the current camera pose to that third pixel is the observation direction of that third pixel.

[0086] During rendering, a pre-defined texture map is typically used to replace the surface of the mesh model. The texture color of a pixel can be obtained from its world coordinates. Because different materials reflect light differently, the actual color of a pixel will change depending on the viewing direction. Therefore, there is a correlation between the direction of the second ray and the color value of the third pixel. The color value of the third pixel can be determined based on its world coordinates and the direction of the second ray, thus determining the target rendered image.

[0087] The target rendered image is the final rendered image in this scheme. The image difference between the target rendered image and the fourth rendered image rendered by the first mesh model meets the preset difference condition. The difference condition can be configured according to actual needs. When the image difference between the target rendered image and the fourth rendered image meets the preset difference condition, it means that the image difference between the target rendered image and the fourth rendered image is small, that is, the image difference between the target rendered image and the high-quality rendered image is small, that is, a high-quality rendered image has been rendered.

[0088] The method provided in this disclosure involves acquiring a second mesh model of the target virtual object and a preset current camera pose; determining the world coordinates and second ray direction of a third pixel based on the current camera pose and the second mesh model, where the second ray direction is the ray direction from the viewpoint under the current camera pose to the third pixel, and the third pixel is a pixel in a third rendered image rendered by the second mesh model under the current camera pose; determining the color value of the third pixel based on the world coordinates and second ray direction to determine the target rendered image, where the image difference between the target rendered image and a fourth rendered image satisfies a preset difference condition, and the fourth rendered image is obtained by rendering a first mesh model under the current camera pose, where the first mesh model is the mesh model of the target virtual object, and the number of triangular faces in the first mesh model is greater than the number of triangular faces in the second mesh model. Based on this solution, high-quality rendered images can be rendered from low-precision 3D models, achieving high-quality 3D rendering and helping to better meet users' needs for high-quality rendered images.

[0089] In one optional method of this disclosure, the color value of the third pixel is determined based on the world coordinates of the third pixel and the direction of the second ray, including:

[0090] The world coordinates of the third pixel and the direction of the second ray are input into the rendering network to obtain the color value of the third pixel output by the rendering network. The rendering network is pre-trained using the above-mentioned rendering network training method.

[0091] In this embodiment of the disclosure, the above-described training method for the rendering network can be used to pre-train the rendering network to predict the target rendering image, so that the image difference between the target rendering image and the fourth rendering image is small.

[0092] Specifically, the world coordinates of the third pixel and the current camera orientation can be input into the rendering network to obtain the color value of the third pixel output by the rendering network.

[0093] As an example, the world coordinates of the third pixel and the unit vector of the current camera direction can be used as inputs to the rendering network, and the RGB value of the color of the third pixel can be used as the output of the rendering network.

[0094] In this embodiment of the disclosure, by using a rendering network, it is possible to predict high-quality rendered images based on a low-precision second mesh model, thereby reducing the performance requirements of the terminal device's GPU and enabling the rendering of high-quality rendered images on terminal devices with weak GPU performance.

[0095] In this solution, a low-precision second mesh model is used during rendering. The data volume of the second mesh model is not too large, which facilitates data transmission and greatly reduces the time required for terminal devices to download materials. Furthermore, the computing power required to predict high-quality rendered images based on the rendering network in this solution is also significantly reduced compared to directly rendering high-precision 3D models.

[0096] In one alternative embodiment of this disclosure, before inputting the world coordinates of the third pixel and the direction of the second ray into the rendering network, the method further includes:

[0097] Obtain the network weight coefficients corresponding to the rendering network;

[0098] The network weights are configured to the pre-deployed local network to obtain the rendering network.

[0099] In this embodiment of the disclosure, the network weight coefficients are the network weight coefficients of the rendering network after training using the above-described rendering network training method.

[0100] When deploying a rendering network to a mobile device, network weight coefficients can be obtained and configured into a pre-deployed local network to obtain the rendering network.

[0101] In one alternative embodiment of this disclosure, the method further includes:

[0102] Receive a rendering request for the target virtual object. The rendering request carries the second mesh model of the target virtual object and the network weight coefficients corresponding to the rendering network.

[0103] In this embodiment of the disclosure, when a mobile terminal receives a rendering request for a target virtual object, it can obtain the second mesh model of the target virtual object and the network weight coefficients corresponding to the rendering network carried in the rendering request, and use them for high-quality rendering of the target virtual object.

[0104] In this embodiment, the rendering network for the target virtual object can be pre-trained on the server side to obtain network weight parameters. When rendering of the target virtual object is required, the second mesh model of the target virtual object and the network weight coefficients are sent to the mobile terminal. The mobile terminal can then achieve high-quality rendering of the target virtual object based on the second mesh model and the network weight coefficients. This ensures that the amount of data to be transmitted between the server and the mobile terminal is not too large, facilitates data transmission, and significantly reduces the time required for the terminal device to download materials.

[0105] In one optional embodiment of this disclosure, the third rendered image is a depth map. Based on the current camera pose and the second mesh model, the world coordinates of the third pixel are determined, including:

[0106] Render the second mesh model under the current camera pose to obtain the third rendered image;

[0107] Obtain the depth value of the third pixel from the third rendered image;

[0108] The world coordinates of the third pixel are determined based on the depth value of the third pixel and the current camera pose.

[0109] In this embodiment of the disclosure, the third rendered image is a depth map, that is, a depth map of the second mesh model rendered under the current camera pose. Then, the depth map can be traversed to obtain the depth value of each third pixel.

[0110] In this embodiment of the disclosure, the world coordinates of the third pixel can be determined based on the depth value of the third pixel and the current camera pose. Specifically, a unit vector from the viewpoint in the camera pose to the third pixel can be found. Multiplying the unit vector by the depth value of the third pixel yields the vector from the viewpoint to the third pixel. Adding this vector to the actual position of the camera gives the world coordinates of the third pixel.

[0111] In this embodiment, image rendering on the mobile terminal can employ deferred rendering, where depth testing is performed first, followed by shading calculation. In this solution, depth testing is performed first (i.e., rendering a second image to obtain depth values), and then the rendering network in this solution predicts the rendered image (i.e., shading calculation is performed). Because this solution eliminates the need for high-precision model rendering and uses deferred rendering, it avoids the large amount of invalid rendering that occurs with forward rendering, thus saving resources on the mobile terminal.

[0112] In one alternative embodiment of this disclosure, the second mesh model is obtained by subtracting the surface area from the first mesh model.

[0113] In this embodiment of the disclosure, the second mesh model can be obtained by reducing the surface area of ​​the first mesh model.

[0114] As an example, the face reduction process in this case is implemented using the QEM model simplification algorithm.

[0115] Based on and Figure 1 The method shown follows the same principle. Figure 3 A schematic diagram of the structure of a training apparatus for a rendering network provided in an embodiment of this disclosure is shown, as follows: Figure 3 As shown, the training device 30 for the rendering network may include:

[0116] The color value acquisition module 310 is used to acquire the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image.

[0117] The world coordinates and ray direction determination module 320 is used to determine the world coordinates and the first ray direction of the second pixel based on the sample camera pose and the second mesh model. The first ray direction is the ray direction from the viewpoint under the sample camera pose to the second pixel. The second pixel is the pixel in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangular faces in the first mesh model is greater than the number of triangular faces in the second mesh model.

[0118] The rendering network training module 330 is used to train the rendering network by taking the world coordinates of the second pixel and the direction of the first ray as input and the color value of the first pixel as output.

[0119] The apparatus provided in this disclosure acquires the color values ​​of each first pixel in a first rendered image. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses, and the first rendered image is a color image. Based on the sample camera poses and a second mesh model, the world coordinates and a first ray direction of a second pixel are determined. The first ray direction is the ray direction from the viewpoint under the sample camera poses to the second pixel. The second pixel is a pixel in the second rendered image rendered by the second mesh model under the sample camera poses. Both the first and second mesh models are mesh models of the target virtual object, and the number of triangular faces in the first mesh model is greater than the number of triangular faces in the second mesh model. The world coordinates and the first ray direction of the second pixel are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieve high-quality 3D rendering, and help better meet users' needs for high-quality rendered images.

[0120] Optionally, the second rendered image is a depth map. When determining the world coordinates and ray direction of the second pixel based on the sample camera pose and the second mesh model, the module is specifically used for:

[0121] The second mesh model is rendered under the sample camera pose to obtain the second rendered image;

[0122] Obtain the depth value of the second pixel from the second rendered image;

[0123] The world coordinates of the second pixel are determined based on the depth value of the second pixel and the pose of the sample camera.

[0124] Optionally, the above-mentioned device further includes:

[0125] The first rendering image processing module is used to render the first mesh model based on a preset texture map and a preset shader under multiple sample camera poses before obtaining the color value of each first pixel in the first rendering image, so as to obtain the first rendering image.

[0126] Optionally, the second mesh model is obtained by subtracting the surface area from the first mesh model.

[0127] It is understood that the above-described modules of the training apparatus for the rendering network in the embodiments of this disclosure have the ability to implement... Figure 1The embodiments shown illustrate the functionality of the corresponding steps in the training method for the rendering network. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the above-described rendering network training apparatus, please refer to [link to relevant documentation]. Figure 1 The corresponding description of the training method of the rendering network in the illustrated embodiment will not be repeated here.

[0128] Based on and Figure 2 The method shown follows the same principle. Figure 4 A schematic diagram of the structure of a rendering apparatus provided in an embodiment of this disclosure is shown, such as... Figure 4 As shown, the rendering device 40 may include:

[0129] The rendering resource acquisition module 410 is used to acquire the second mesh model of the target virtual object and the preset current camera pose;

[0130] The world coordinates and ray direction determination module 420 is used to determine the world coordinates and second ray direction of the third pixel based on the current camera pose and the second mesh model. The second ray direction is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is the pixel in the third rendered image rendered by the second mesh model under the current camera pose.

[0131] The rendering module 430 is used to determine the color value of the third pixel based on the world coordinates of the third pixel and the direction of the second ray, so as to determine the target rendering image. The image difference between the target rendering image and the fourth rendering image meets the preset difference condition. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model.

[0132] The apparatus provided in this disclosure acquires a second mesh model of a target virtual object and a preset current camera pose. Based on the current camera pose and the second mesh model, it determines the world coordinates and second ray direction of a third pixel. The second ray direction is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is a pixel in a third rendered image rendered by the second mesh model under the current camera pose. Based on the world coordinates and second ray direction of the third pixel, it determines the color value of the third pixel to determine the target rendered image. The image difference between the target rendered image and the fourth rendered image satisfies a preset difference condition. The fourth rendered image is obtained by rendering a first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object, and the number of triangular faces in the first mesh model is greater than the number of triangular faces in the second mesh model. Based on this solution, a high-quality rendered image can be rendered from a low-precision 3D model, achieving high-quality 3D rendering and helping to better meet users' needs for high-quality rendered images.

[0133] Optionally, when determining the color value of the third pixel based on its world coordinates and the direction of the second ray, the rendering module specifically uses the following methods:

[0134] The world coordinates of the third pixel and the direction of the second ray are input into the rendering network to obtain the color value of the third pixel output by the rendering network. The rendering network is pre-trained using the above-mentioned rendering network training method.

[0135] Optionally, the above apparatus further includes a rendering network deployment module, the rendering network deployment module being used for:

[0136] Before inputting the world coordinates of the third pixel and the direction of the second ray into the rendering network, obtain the network weight coefficients corresponding to the rendering network;

[0137] The network weight coefficients are configured into a pre-deployed local network to obtain the rendering network.

[0138] Optionally, the above-mentioned device further includes:

[0139] The rendering request receiving module is used to receive rendering requests for the target virtual object. The rendering request carries the second mesh model of the target virtual object and the network weight coefficients corresponding to the rendering network.

[0140] Optionally, the third rendered image is a depth map. When determining the world coordinates and ray direction of the third pixel based on the current camera pose and the second mesh model, the module is specifically used for:

[0141] Render the second mesh model under the current camera pose to obtain the third rendered image;

[0142] Obtain the depth value of the third pixel from the third rendered image;

[0143] The world coordinates of the third pixel are determined based on the depth value of the third pixel and the current camera pose.

[0144] Optionally, the second mesh model is obtained by subtracting the surface area from the first mesh model.

[0145] It is understood that the above-described modules of the rendering apparatus in the embodiments of this disclosure have the ability to implement... Figure 2 The rendering method in the illustrated embodiment demonstrates the functionality of corresponding steps. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. These modules can be software and / or hardware; each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module in the rendering apparatus, please refer to [link to relevant documentation]. Figure 2 The corresponding descriptions of the rendering methods in the embodiments shown are not repeated here.

[0146] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0148] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training or rendering method for a rendering network as provided in the embodiments of this disclosure.

[0149] Compared with existing technologies, this electronic device obtains the color values ​​of each first pixel in a first rendered image. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses and is a color image. Based on the sample camera poses and a second mesh model, it determines the world coordinates and first ray direction of the second pixel. The first ray direction is the ray direction from the viewpoint under the sample camera poses to the second pixel. The second pixel is a pixel in the second rendered image rendered by the second mesh model under the sample camera poses. Both the first and second mesh models are mesh models of the target virtual object, with the first mesh model having a greater number of triangular faces than the second mesh model. The world coordinates and first ray direction of the second pixel are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieving high-quality 3D rendering and helping to better meet users' needs for high-quality rendered images.

[0150] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform a training or rendering method for a rendering network as provided in the embodiments of this disclosure.

[0151] Compared with existing technologies, this readable storage medium obtains the color values ​​of each first pixel in a first rendered image. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses and is a color image. Based on the sample camera poses and a second mesh model, the world coordinates and first ray direction of the second pixel are determined. The first ray direction is the ray direction from the viewpoint under the sample camera poses to the second pixel. The second pixel is a pixel in the second rendered image rendered by the second mesh model under the sample camera poses. Both the first and second mesh models are mesh models of the target virtual object, with the first mesh model having a greater number of triangular faces than the second mesh model. The world coordinates and first ray direction of the second pixel are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieving high-quality 3D rendering and helping to better meet users' needs for high-quality rendered images.

[0152] The computer program product includes a computer program that, when executed by a processor, implements a method for training or rendering a rendering network as provided in embodiments of this disclosure.

[0153] Compared with existing technologies, this computer program product obtains the color values ​​of each first pixel in a first rendered image. The first rendered image is obtained by rendering a first mesh model under multiple preset sample camera poses and is a color image. Based on the sample camera poses and a second mesh model, it determines the world coordinates and first ray direction of the second pixel. The first ray direction is the ray direction from the viewpoint under the sample camera poses to the second pixel. The second pixel is a pixel in the second rendered image rendered by the second mesh model under the sample camera poses. Both the first and second mesh models are mesh models of the target virtual object, with the first mesh model having a greater number of triangular faces than the second mesh model. The world coordinates and first ray direction of the second pixel are used as input, and the color value of the first pixel is used as output to train the rendering network. The rendering network trained based on this scheme can be used to predict high-quality rendered images, achieving high-quality 3D rendering and helping to better meet users' needs for high-quality rendered images.

[0154] Figure 5 A schematic block diagram of an example electronic device 50 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] like Figure 5 As shown, the electronic device 50 includes a computing unit 510, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 520 or a computer program loaded from a storage unit 580 into a random access memory (RAM) 530. The RAM 530 may also store various programs and data required for the operation of the device 50. The computing unit 510, ROM 520, and RAM 530 are interconnected via a bus 540. An input / output (I / O) interface 550 is also connected to the bus 540.

[0156] Multiple components in device 50 are connected to I / O interface 550, including: input unit 560, such as keyboard, mouse, etc.; output unit 570, such as various types of monitors, speakers, etc.; storage unit 580, such as disk, optical disk, etc.; and communication unit 590, such as network card, modem, wireless transceiver, etc. Communication unit 590 allows device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] The computing unit 510 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 510 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 510 executes the training or rendering methods of the rendering network provided in the embodiments of this disclosure. For example, in some embodiments, executing the training or rendering methods of the rendering network provided in the embodiments of this disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 580. In some embodiments, part or all of the computer program can be loaded and / or installed on device 50 via ROM 520 and / or communication unit 590. When the computer program is loaded into RAM 530 and executed by computing unit 510, one or more steps of the training or rendering methods of the rendering network provided in the embodiments of this disclosure can be performed. Alternatively, in other embodiments, the computing unit 510 may be configured by any other suitable means (e.g., by means of firmware) to perform the training or rendering methods of the rendering network provided in the embodiments of this disclosure.

[0158] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0162] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0163] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0164] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a rendering network, comprising: Obtain the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image. Based on the sample camera pose and the second mesh model, the world coordinates of the second pixel and the direction of the first ray are determined. The first ray direction is the ray direction from the viewpoint under the sample camera pose to the second pixel. The second pixel is the pixel in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model. The world coordinates of the second pixel and the direction of the first ray are used as inputs, and the color value of the first pixel is used as output to train the rendering network.

2. The method according to claim 1, wherein, The second rendered image is a depth map. The step of determining the world coordinates of the second pixel based on the sample camera pose and the second mesh model includes: The second mesh model is rendered under the sample camera pose to obtain the second rendered image; Obtain the depth value of the second pixel from the second rendered image; The world coordinates of the second pixel are determined based on the depth value of the second pixel and the pose of the sample camera.

3. The method according to claim 1 or 2, wherein before obtaining the color value of each first pixel in the first rendered image, the method further comprises: The first mesh model is rendered using a preset texture map and a preset shader under multiple sample camera poses to obtain the first rendered image.

4. The method according to any one of claims 1-3, wherein, The second mesh model is obtained by reducing the surface area of ​​the first mesh model.

5. A rendering method, comprising: Obtain the second mesh model of the target virtual object and the preset current camera pose; Based on the current camera pose and the second mesh model, the world coordinates and the direction of the second ray of the third pixel are determined. The direction of the second ray is the current ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is the pixel in the third rendered image rendered by the second mesh model under the current camera pose. The world coordinates of the third pixel and the direction of the second ray are input into the rendering network to obtain the color value of the third pixel output by the rendering network, so as to determine the target rendering image. The image difference between the target rendering image and the fourth rendering image satisfies a preset difference condition. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model. The rendering network is pre-trained by the rendering network training method of any one of claims 1-4.

6. The method according to claim 5, wherein before inputting the world coordinates of the third pixel and the direction of the second ray into the rendering network, the method further comprises: Obtain the network weight coefficients corresponding to the rendering network; The network weight coefficients are configured into a pre-deployed local network to obtain the rendering network.

7. The method according to claim 6, further comprising: Receive a rendering request for a target virtual object, the rendering request carrying a second mesh model of the target virtual object and the network weight coefficients corresponding to the rendering network.

8. The method according to any one of claims 5-7, wherein, The third rendered image is a depth map. Determining the world coordinates of the third pixel based on the current camera pose and the second mesh model includes: The second mesh model is rendered under the current camera pose to obtain the third rendered image; Obtain the depth value of the third pixel from the third rendered image; The world coordinates of the third pixel are determined based on the depth value of the third pixel and the current camera pose.

9. The method according to any one of claims 5-7, wherein, The second mesh model is obtained by reducing the surface area of ​​the first mesh model.

10. A training apparatus for rendering a network, comprising: The color value acquisition module is used to acquire the color value of each first pixel in the first rendered image. The first rendered image is obtained by rendering the first mesh model under multiple preset sample camera poses. The first rendered image is a color image. The world coordinates and ray direction determination module is used to determine the world coordinates and first ray direction of the second pixel point based on the sample camera pose and the second mesh model. The first ray direction is the ray direction from the viewpoint under the sample camera pose to the second pixel point. The second pixel point is the pixel point in the second rendered image rendered by the second mesh model under the sample camera pose. Both the first mesh model and the second mesh model are mesh models of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model. The rendering network training module is used to train the rendering network by taking the world coordinates of the second pixel and the direction of the first ray as input and the color value of the first pixel as output.

11. The apparatus according to claim 10, wherein, The second rendered image is a depth map. When determining the world coordinates and ray direction based on the sample camera pose and the second mesh model, the world coordinates and ray direction determination module is specifically used for: The second mesh model is rendered under the sample camera pose to obtain the second rendered image; Obtain the depth value of the second pixel from the second rendered image; The world coordinates of the second pixel are determined based on the depth value of the second pixel and the pose of the sample camera.

12. The apparatus according to claim 10 or 11, further comprising: The first rendering image processing module is used to render the first mesh model based on a preset texture map and a preset shader under multiple sample camera poses before obtaining the color value of each first pixel in the first rendering image, so as to obtain the first rendering image.

13. The apparatus according to any one of claims 10-12, wherein, The second mesh model is obtained by reducing the surface area of ​​the first mesh model.

14. A rendering apparatus, comprising: The rendering resource acquisition module is used to acquire the second mesh model of the target virtual object and the preset current camera pose; The world coordinates and ray direction determination module is used to determine the world coordinates and second ray direction of the third pixel based on the current camera pose and the second mesh model. The second ray direction is the ray direction from the viewpoint under the current camera pose to the third pixel. The third pixel is a pixel in the third rendered image rendered by the second mesh model under the current camera pose. A rendering module is used to input the world coordinates of the third pixel and the direction of the second ray into a rendering network to obtain the color value of the third pixel output by the rendering network, so as to determine a target rendering image. The image difference between the target rendering image and the fourth rendering image satisfies a preset difference condition. The fourth rendering image is obtained by rendering the first mesh model under the current camera pose. The first mesh model is the mesh model of the target virtual object. The number of triangles in the first mesh model is greater than the number of triangles in the second mesh model. The rendering network is pre-trained by the rendering network training method of any one of claims 1-4.

15. The apparatus of claim 14, further comprising a rendering network deployment module, the rendering network deployment module being configured to: Before inputting the world coordinates of the third pixel and the direction of the second ray into the rendering network, obtain the network weight coefficients corresponding to the rendering network; The network weight coefficients are configured into a pre-deployed local network to obtain the rendering network.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images