Material rendering method and device in three-dimensional scene and computing device cluster

Abstract

The invention discloses a material rendering method and device in a three-dimensional scene and a computing device cluster, and belongs to the technical field of computer graphics. The method comprises the steps that after material information of a three-dimensional model and coloring point information of coloring points are obtained, information extraction and compression encoding are conducted on the material information through an encoder of a neural network, the material features of a to-be-rendered material on the coloring points are obtained, and the color points of the to-be-rendered material on the coloring points are obtained through the good expression ability of a deep network; and the output material features contain rich information and have discrimination for texture maps of different materials. Furthermore, through a decoder of the neural network, the material features and the coloring point information are decoded, rendering reference information of the to-be-rendered material, namely relevant parameters of a bidirectional reflectance distribution function (BRDF), is output, in this way, the calculation process of the BRDF is simplified, the relevant parameters of the BRDF can be rapidly and accurately output for various materials, and the calculation efficiency of the BRDF is improved. And high-quality material rendering can be realized.

Description

[0001] This application claims priority to Chinese patent application No. 202311321656.9, filed on October 12, 2023, with invention name “Graphics rendering method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer graphics technology, and in particular to a material rendering method, device and computing device cluster in a three-dimensional scene. Background Art

[0003] With the rapid development of computer graphics technology, more and more materials can be presented in three-dimensional scenes. At present, different materials are usually defined based on bidirectional reflectance distribution function (BRDF).

[0004] In related technologies, most materials such as metal, wall, leather, etc. can be rendered through a BRDF material model based on microfacet theory to obtain highly realistic rendering results. The microfacet theory refers to the assumption that an object is composed of a series of tiny mirror surfaces at the microscopic level, thereby deriving a material model.

[0005] However, for some complex materials, such as cloth, hair, skin, car paint, etc., it is difficult to achieve accurate modeling using the above method, resulting in poor rendering quality of such materials. Summary of the invention

[0006] The embodiments of the present application provide a material rendering method, apparatus, and computing device cluster in a three-dimensional scene, which can quickly and accurately output BRDF-related parameters for various materials to achieve high-quality material rendering.

[0007] In a first aspect, the present application provides a material rendering method in a three-dimensional scene, the method comprising:

[0008] Acquire material information of a three-dimensional model in a three-dimensional scene and shading point information of a shading point on the three-dimensional model, wherein the material information indicates a material texture map and material geometric parameters of the three-dimensional model, and the shading point information indicates a lighting condition of the shading point;

[0009] The material information is encoded by an encoder of a neural network to obtain material features of the material to be rendered at the shading point, wherein the material features are used to characterize texture features and geometric features of the material to be rendered;

[0010] Decoding the material feature and the shading point information through a decoder of the neural network to obtain rendering reference information of the material to be rendered, wherein the rendering reference information indicates a specular reflection parameter and a diffuse reflection parameter of the material to be rendered;

[0011] Rendering the material of the shading point based on the rendering reference information.

[0012] In the above method, after obtaining the material information of the three-dimensional model and the shading point information of the shading point, the material information is extracted and compressed by the encoder of the neural network to obtain the material features of the material to be rendered at the shading point. With the good expression ability of the deep network, the output material features contain rich information and are distinguishable for different material texture maps. Further, the material features and shading point information are decoded by the decoder of the neural network to output the rendering reference information of the material to be rendered, that is, the relevant parameters of the bidirectional reflectance distribution function BRDF. In this way, the calculation process of BRDF is simplified, and the BRDF related parameters can be quickly and accurately output for various materials, so as to achieve high-quality material rendering.

[0013] In some embodiments, the encoder includes a first encoding unit and a second encoding unit; the encoder through the neural network encodes the material information to obtain the material characteristics of the material to be rendered at the shading point, including:

[0014] By means of the first encoding unit, the material texture map is encoded to obtain an intermediate material feature;

[0015] The intermediate material features and the material geometric parameters are encoded by the second encoding unit to obtain the material features.

[0016] In some embodiments, the method further comprises at least one of the following:

[0017] Storing the intermediate material features so as to include the intermediate material features in a material rendering process of other shading points when the material texture map is not updated;

[0018] The material feature is stored so as to participate the material feature in the material rendering process of other shading points when the material information is not updated.

[0019] Different material information is encoded separately by the two encoding units of the encoder, so that the encoder does not need to be executed repeatedly for each frame, and the material texture map may not be completely re-executed when it is updated, thereby effectively reducing the computing overhead and improving rendering efficiency.

[0020] In some embodiments, the decoder includes a first decoding unit and a second decoding unit, the shading point information indicates a light footprint, a light direction, and a sight direction of the shading point, and the light footprint indicates a light distribution on a surface of the shading point; the decoder through the neural network decodes the material features and the shading point information to obtain rendering reference information of the material to be rendered, including:

[0021] Decoding the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature;

[0022] The intermediate decoding feature, the light direction, and the sight direction are decoded by the second decoding unit to obtain the rendering reference information.

[0023] In some embodiments, the method further comprises:

[0024] Encoding the light footprint to obtain the encoded light footprint to expand the data length of the light footprint;

[0025] The decoding of the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature includes:

[0026] The encoded light footprint and the material feature are decoded by the first decoding unit to obtain the intermediate decoded feature.

[0027] It should be understood that since the textures corresponding to different detail levels are usually determined by the light footprint at the shading point, through the above method, a first decoding unit is used to decode the light footprint and material features, so that the decoder can better distinguish and decode the texture features of each detail level, so that the material rendering method provided in the present application can support material rendering at multiple detail levels.

[0028] In some embodiments, rendering the material of the shading point based on the rendering reference information includes:

[0029] Determine a function value of a bidirectional reflectance distribution function of the shading point based on the rendering reference information, the specular reflection color and the diffuse reflection color of the shading point;

[0030] Based on the function value, the material of the shading point is rendered.

[0031] In some embodiments, the obtaining of material information of a three-dimensional model in a three-dimensional scene and shading point information of shading points on the three-dimensional model includes:

[0032] The material information and the shading point information are obtained from a rendering pipeline of the three-dimensional model.

[0033] In some embodiments, the material to be rendered is a cloth material;

[0034] The material texture map includes a normal map and a direction map;

[0035] The material geometric parameters include warp yarn parameters and weft yarn parameters.

[0036] In some embodiments, the training process of the neural network includes:

[0037] In an i-th iteration of training the neural network, sample material information and sample shading point information of a sample shading point are input into the neural network to obtain predicted rendering reference information of the sample shading point, wherein the predicted rendering reference information includes predicted specular reflection parameters and predicted diffuse reflection parameters of a material to be rendered at the sample shading point, where i is a positive integer;

[0038] Determine a loss value based on the predicted rendering reference information and the annotated rendering reference information of the sample shading point, wherein the annotated rendering reference information includes an annotated specular reflection parameter and an annotated diffuse reflection parameter of a material to be rendered at the sample shading point;

[0039] Based on the loss value, the network parameters of the neural network are adjusted, and based on the adjusted neural network, an i+1th iteration is performed.

[0040] In some embodiments, determining the loss value based on the predicted rendering reference information and the labeled rendering reference information of the sample shading point includes:

[0041] Determining a first loss value based on the predicted specular reflection parameter and the annotated specular reflection parameter;

[0042] Determining a second loss value based on the predicted diffuse reflection parameter and the marked diffuse reflection parameter;

[0043] A weighted sum is performed on the first loss value and the second loss value to obtain the loss value.

[0044] In this way, since the learning difficulty of specular reflection parameters is usually greater than that of diffuse reflection parameters, this method can improve the training effect of the neural network.

[0045] In a second aspect, an embodiment of the present application provides a device for rendering materials in a three-dimensional scene, the device comprising at least one functional module for executing a method for rendering materials in a three-dimensional scene as provided in the first aspect or any possible implementation of the first aspect.

[0046] In a third aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs a material rendering method in a three-dimensional scene provided by the aforementioned first aspect or any possible implementation of the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the material rendering method in a three-dimensional scene provided by the aforementioned first aspect or any possible implementation of the first aspect. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computing device cluster, enables the computing device cluster to implement the material rendering method in a three-dimensional scene provided by the aforementioned first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when it is necessary to implement the aforementioned material rendering method in a three-dimensional scene, the computer program product may be downloaded and executed on a computing device or a computing device cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0050] Figure 2 is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application;

[0053] Figure 5 is a flow chart of a material rendering method in a three-dimensional scene provided in an embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the architecture of a neural network provided in an embodiment of the present application;

[0055] Figure 7 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0056] Figure 8 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0057] Fig. 9 is a schematic diagram of a material rendering method in a three-dimensional scene provided in an embodiment of the present application;

[0058] Fig.10 It is a structural schematic diagram of a material rendering device in a three-dimensional scene provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the three-dimensional models, neural networks, etc. involved in this application are all obtained with full authorization.

[0060] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0061] The rendering pipeline, also known as the rendering pipeline, is the process of generating or drawing a two-dimensional image under many conditions such as a virtual camera, a three-dimensional object, a light source, a lighting mode, and a texture. Currently, commonly used rendering pipelines include rasterization rendering pipelines, ray tracing rendering pipelines, and hybrid pipelines.

[0062] Bidirectional reflectance distribution function (BRDF) is a four-dimensional function that defines the reflection of light on an opaque surface. It describes the distribution of incident light in various outgoing directions after being reflected by a certain material surface, or in other words, how light and objects interact. In the field of computer graphics technology, different materials are usually defined based on BRDF. For example, the BRDF parameters corresponding to the shading point on a cup in a three-dimensional scene, including specular reflection parameters and diffuse reflection parameters, are calculated to obtain the function value of the BRDF of the shading point; based on the function value, the material of the shading point is rendered so that the cup shows a certain degree of specular reflection and diffuse reflection, giving people a ceramic feeling.

[0063] Material is a collection of properties that describe the appearance and optical properties of an object. It includes the object's color, reflection properties (such as diffuse reflection, specular reflection), transparency, refractive index, etc. The material defines how the object interacts with light and determines the appearance of the object when rendered.

[0064] Pixel is the smallest grid unit in a two-dimensional grid image.

[0065] Texture is an image used to simulate the details and texture of an object's surface. It can contain color information, detailed patterns, texture details, etc. By mapping the texture to the model surface, the model can be given a more realistic appearance and details.

[0066] Texel, short for texture element, is the basic unit in the texture space of computer graphics. Schematically, an image is composed of an arrangement of pixels, and a texture is represented by an arrangement of texels.

[0067] A texture is a two-dimensional multi-channel grid image composed of texels. Each texel has its discrete two-dimensional coordinates and stores a multi-dimensional vector, which is used to record the surface features of a three-dimensional object. When the vertices of a three-dimensional object's geometric triangles contain their two-dimensional coordinates in the texture, the two-dimensional coordinates of the shading point in the texture can be obtained through interpolation technology, and then the corresponding texture can be calculated. In the rendering process, materials and textures are used in conjunction with each other. Materials define the properties of objects, and texture maps give models realism and detail effects by providing specific color and texture information.

[0068] Shading point. In the raster process, a shading point refers to the point on the surface of an object corresponding to a pixel whose color is to be drawn; in the ray tracing process, a shading point refers to the intersection of the light ray and the surface of the object.

[0069] Ray tracing is a rendering method in the field of computer graphics technology. Its main idea is to emit light from the viewpoint to the pixels on the imaging plane, find the intersection point of the nearest object intersecting with the light, and continue to trace another light from the intersection point in the direction of reflection or refraction, and so on recursively until the light escapes the scene or reaches the light source.

[0070] The application scenarios and implementation environment of this application are introduced below.

[0071] The technical solution provided in the present application can be applied to the material rendering stage for a three-dimensional model in a three-dimensional scene, that is, the material rendering stage for the shading points on the three-dimensional model, and is suitable for the gaming field, VR field, digital human field, etc., wherein the material can be conventional materials such as metal, wall skin, leather, or complex materials such as cloth, hair, skin, car paint, etc., and the present application does not limit this.

[0072] Figure 1 Schematic diagram of an implementation environment provided by the embodiment of the present application. Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102, and the terminal 101 is directly or indirectly connected to the server 102 via a wireless network or a wired network.

[0073] Terminal 101 has a three-dimensional model rendering display function, and can display the rendering of a three-dimensional model in a three-dimensional scene, and display the corresponding rendering results. Terminal 101 is at least one of a smart phone, a game console, a desktop computer, an augmented reality terminal, a tablet computer, an e-book reader, and a laptop. Terminal 101 installs and runs an application that supports the three-dimensional model rendering display function. The application can be a client, a browser, etc., and the present application is not limited to this. Taking the application as a client as an example, the user can input relevant information of the three-dimensional model in the three-dimensional scene through the client, triggering terminal 101 to render the three-dimensional model, and display the corresponding rendering results on the client interface.

[0074] The server 102 is used to provide background services for the application programs running on the terminal 101. The server 102 is an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Take the server 102 as an example of a cloud server, and a cloud server refers to a service based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data calculation process is processed and analyzed at the remote end and returned to the user, with the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud server can realize the rapid issuance and release of configurable computing resources with a small management cost, or a low interaction complexity between the user and the service provider. In some embodiments, the cloud server can also be a virtual machine instance, a container instance, etc., and the present application is not limited thereto.

[0075] In an embodiment of the present application, terminal 101 can independently execute the material rendering method in the three-dimensional scene involved in the following method embodiment, or can work in collaboration with server 102. For example, terminal 101 sends the acquired relevant information of the three-dimensional model to server 102, and server 102 renders the three-dimensional model to obtain the rendering result, and feeds the rendering result back to terminal 101, and terminal 101 displays the rendering result. This application does not limit this.

[0076] It should be understood that the terminal 101 may refer to one of multiple terminals or a set of multiple terminals; the server 102 may refer to one of multiple servers or a set of multiple servers. The embodiment of the present application does not limit the number and type of each device in the implementation environment.

[0077] In some embodiments, the wireless network or wired network described above uses standard communication technology and / or protocol. The network includes, but is not limited to, any combination of a data center network, a storage area network (SAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some implementations, the data exchanged through the network is represented by technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies such as secure socketslayer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can be used to encrypt all or part of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0078] The hardware structure of the terminal and server in the above implementation environment is introduced below.

[0079] The present application embodiment provides a computing device that can be configured as the above-mentioned terminal or server. Figure 2 , Figure 2 Schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 200 includes a memory 201, a processor 202, a communication interface 203 and a bus 204. The memory 201, the processor 202 and the communication interface 203 are connected to each other through the bus 204.

[0080] The memory 201 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. Schematically, the memory 201 is used to store at least one section of program code, and when the program code stored in the memory 201 is executed by the processor 202, the processor 202 is used to execute the steps performed by the computing device in the following method embodiment.

[0081] The processor 202 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling the execution of the program of the present application. The processor 202 may be a single-CPU processor or a multi-CPU processor. The number of processors 202 may be one or more. The communication interface 203 uses a transceiver module such as a transceiver to implement communication between the computing device 200 and other devices or communication networks. For example, data may be acquired through the communication interface 203.

[0082] The memory 201 and the processor 202 may be separately provided or integrated together.

[0083] Bus 204 may include a pathway for transmitting information between various components of computing device 200 (eg, memory 201 , processor 202 , communication interface 203 ).

[0084] The present application also provides a computing device cluster, which includes multiple computing devices. The multiple computing devices include a server and a terminal, wherein the server can be a server, such as a central server, an edge server, or a local server in a local data center, and the terminal can be a desktop computer, a laptop computer, or a smart phone. Figure 3 , Figure 3 Schematic diagram of a computing device cluster provided in an embodiment of the present application. Figure 3 As shown, the computing device cluster includes multiple computing devices 200. The same instructions for executing the material rendering method in the three-dimensional scene may be stored in the memory 201 of the multiple computing devices 200 in the computing device cluster. In some embodiments, the memory 201 of the multiple computing devices 200 in the computing device cluster may also respectively store partial instructions for executing the material rendering method in the three-dimensional scene. In other words, the combination of multiple computing devices 200 is used to jointly execute the instructions of the material rendering method in the three-dimensional scene.

[0085] In some embodiments, multiple computing devices 200 in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 4 Schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application. Figure 4 As shown, two computing devices 200 are connected via a network. Specifically, the network is connected via the communication interface 203 in each computing device 200. It should be understood that Figure 4 The functionality of the computing device 200 shown in FIG. 2 may also be performed by multiple computing devices 200 .

[0086] The following is an introduction to the material rendering method in the three-dimensional scene provided by this application.

[0087] The material rendering method in the three-dimensional scene provided by the present application provides a method for outputting the BRDF related parameters of the material to be rendered at the shading point through a neural network, which simplifies the BRDF calculation process, and can quickly and accurately output the BRDF related parameters for various materials, so as to achieve high-quality material rendering. Figure 5 , this method is introduced.

[0088] Figure 5 is a flow chart of a material rendering method in a three-dimensional scene provided in an embodiment of the present application. Figure 5 As shown, the method is introduced by taking the execution of a computing device as an example, for example, by a graphics processor GPU in the computing device. The method can be applied to the real-time rendering process of a three-dimensional model in a three-dimensional scene, including the following steps 501 to 504.

[0089] 501. A computing device obtains material information of a three-dimensional model in a three-dimensional scene and shading point information of shading points on the three-dimensional model, wherein the material information indicates a material texture map and material geometric parameters of the three-dimensional model, and the shading point information indicates a lighting condition of the shading point.

[0090] In the embodiments of the present application, a 3D scene refers to a 3D virtual scene, such as a virtual game space. A 3D model refers to an object to be rendered in a 3D scene, such as a 3D model of a digital person's clothing, curtains in a virtual home space, a virtual racing car, etc., but the present application is not limited thereto. It should be understood that a 3D model usually includes multiple shading points. In the present embodiment, any shading point of a 3D model is used as an example for introduction. The material information and shading point information of the 3D model are introduced below.

[0091] The material information of the three-dimensional model indicates the material texture map and material geometric parameters of the three-dimensional model, wherein the material texture map is used to characterize the details and texture of the surface material of the three-dimensional model, for example, the material texture map includes at least one of the following: a normal map, a direction map, a roughness map, and a height field scaling map, etc. The material geometric parameters are used to characterize the geometric structure of the three-dimensional surface material. It should be understood that different materials often have different geometric structures. Taking the cloth material as an example, since the cloth material is a complex structure of warp and weft yarns interwoven, the material geometric parameters of the cloth material may include warp yarn parameters and weft yarn parameters, for example, including the roughness of the warp and weft yarns and the height field scaling ratio. For another example, taking the hair material as an example, its material geometric parameters include longitudinal axis parameters and transverse axis parameters, etc. In some embodiments, the computing device obtains the material information of the three-dimensional model from the rendering pipeline of the three-dimensional model. In the process of reading the three-dimensional scene data, the computing device may obtain the material texture map from it, perform sampling operations on the material texture map, and obtain the material geometric parameters.

[0092] The shading point information of the shading point on the three-dimensional model indicates the lighting conditions of the shading point. It should be understood that shading is a local process for the shading point, and the key is to calculate the reflection of light on the specified shading point. For example, the shading point information includes at least one of the following: the light footprint of the shading point, the light direction, and the line of sight direction. Among them, the light footprint indicates the distribution of light on the surface of the shading point, such as the position and size of the light. In some embodiments, if the computing device uses a ray tracing rendering pipeline to render the three-dimensional model, the computing device can quickly obtain the shading point information of the shading point from the rendering pipeline. Of course, if the computing device uses other rendering pipelines (such as a rasterization rendering pipeline), the computing device obtains data related to the shading point information from the rendering pipeline, and after processing these data, the shading point information is obtained. This application does not limit the method for obtaining the shading point information. In addition, the shading point information can also include other information such as shading point coordinates, which is not limited in this application.

[0093] 502. The computing device encodes the material information through the encoder of the neural network to obtain the material features of the material to be rendered at the shading point, where the material features are used to characterize the texture features and geometric features of the material to be rendered.

[0094] In an embodiment of the present application, a neural network refers to a neural network obtained through training based on an artificial intelligence network, and the neural network includes an encoder and a decoder, wherein the output layer of the encoder is connected to the input layer of the decoder. Among them, the encoder is used to encode material information to obtain material features of the material to be rendered at the shading point, or in other words, the encoder extracts and compresses information from its input data, and has the function of assisting the decoder to analyze the material in advance, that is, by extracting the texture features and geometric features of the material to be rendered, combining and understanding them, and converting them into coding results that are easily extracted and analyzed by the decoder and compressing them, so as to obtain the material features of the material to be rendered. The decoder is used to decode the material features and shading point information to obtain rendering reference information of the material to be rendered, and the rendering reference information indicates the specular reflection parameters and diffuse reflection parameters of the parameters to be rendered, that is, the parameters of the bidirectional reflectance distribution function BRDF. For example, reference Figure 6 , Figure 6 Schematic diagram of a neural network architecture provided by an embodiment of the present application. Figure 6 As shown, the neural network includes an encoder 601 and a decoder 602, wherein the output layer of the encoder 601 is connected to the input layer of the decoder 602, the material information is input into the encoder 601 to obtain the material features of the material to be rendered at the shading point, the material features and the shading point information of the shading point are input into the decoder 602 to obtain the rendering reference information of the material to be rendered.

[0095] In some embodiments, the encoder of the neural network includes a first encoding unit and a second encoding unit. The two encoding units encode the material texture map and the material geometric parameters indicated by the material information respectively to obtain corresponding features. Schematically, this step 502 includes the following steps:

[0096] Step A1: Encode the material texture map through the first encoding unit to obtain intermediate material features.

[0097] In some embodiments, the computing device stores the intermediate material features so that the intermediate material features can be included in the material rendering process of other shading points when the material texture map is not updated. It should be understood that since the material texture map is encoded by the first encoding unit of the encoder, it is decoupled from the encoding process of the material geometric parameters. In this way, after the intermediate material features are stored, if the material texture map is not updated during the rendering of the material of other shading points, the computing device can perform subsequent steps based on the stored intermediate material features without running the first encoding unit again, thereby reducing the computing overhead and improving the rendering efficiency.

[0098] Step A2: Encode the intermediate material features and material geometric parameters through the second encoding unit to obtain material features.

[0099] In some embodiments, the computing device stores material features so that the material features can be included in the material rendering process of other shading points when the material information is not updated. It should be understood that after the material features are stored, if the material texture map and material geometry parameters are not updated during the rendering of the material of other shading points, the computing device can perform subsequent steps based on the stored material features without running the encoder again, thereby reducing computing overhead and improving rendering efficiency.

[0100] It can be seen that different material information is encoded respectively by the two encoding units of the encoder, so that the encoder does not need to be executed repeatedly for each frame, and the material texture map may not be completely re-executed when it is updated, thereby effectively reducing the computing overhead and improving rendering efficiency.

[0101] Reference below Figure 7 , the encoder structure of the above neural network is illustrated with examples. Figure 7 Schematic diagram of the structure of an encoder provided in an embodiment of the present application. Figure 7As shown, the encoder includes a first encoding unit 701 and a second encoding unit 702. The first encoding unit 701 is a thirteen-layer convolutional neural network. Except for the first and last layers, residual connections are added between every two layers to further improve the network expression capability. Taking the material texture map size of 512×512 as an example, after the material texture map is input into the first encoding unit 701, the first encoding unit 701 outputs a 64-channel intermediate material feature, wherein the processing operations of each layer in the first encoding unit 701 are all examples and do not constitute a limitation of the present application. The second encoding unit 702 is a four-layer multi-layer perceptron. Similar to the first encoding unit 701, except for the first and last layers, residual connections are added between every two layers to further improve the network expression capability. Taking the material geometry parameter as a 4-channel input data as an example, the material geometry parameter and the intermediate material feature are input into the second encoding unit 702, and the second encoding unit 702 outputs a 64-channel material feature. Combined with Figure 7 It can be seen that the first encoding unit and the second encoding unit form a relatively deep network under the requirements of real-time rendering, which can process two types of inputs in combination. Moreover, since the ability of deep networks to extract features is stronger than that of shallow networks, such a deep network can enhance the processing power of the encoder, making the feature information of the encoded compression more representative and easier to be decoded by the subsequent decoder.

[0102] In addition, the above Figure 7 The encoder structure shown is for illustrative purposes only and does not constitute a limitation on the encoder structure in the present application. Any other network based on machine learning or deep learning and intended to obtain material characteristics can be used as an encoder for the neural network in the embodiments of the present application.

[0103] 503. The computing device decodes the material features and shading point information through a decoder of a neural network to obtain rendering reference information of the material to be rendered, where the rendering reference information indicates specular reflection parameters and diffuse reflection parameters of the material to be rendered.

[0104] In the embodiment of the present application, based on the aforementioned step 502, it can be known that the specular reflection parameters and diffuse reflection parameters of the material to be rendered are also the parameters of BRDF, which will not be described in detail here.

[0105] In some embodiments, the decoder of the neural network includes a first decoding unit and a second decoding unit. The two decoding units are used to combine and decode the material features and the shading point information to obtain rendering reference information. Schematically, taking the shading point information including the light footprint, light direction, and sight direction of the shading point as an example, this step 503 includes the following steps:

[0106] Step B1: Decode the light footprint and material features through the first decoding unit to obtain intermediate decoding features.

[0107] In some embodiments, the computing device encodes the light footprint to obtain the encoded light footprint to expand the data length of the light footprint. Accordingly, this step includes: decoding the encoded light footprint and material features through a first decoding unit to obtain intermediate decoded features. It should be understood that since the data lengths of light footprints and material features are often quite different, encoding the light footprint to expand its data length can reduce the difficulty of the network to express the light footprint, enhance the expression ability of the neural network, and provide a basis for outputting more accurate decoding results. For example, taking the input data of 3 channels of light footprint as an example, by adding One-blob encoding (a coding method for improving the completion ability and consistency of unobserved areas) to each channel of the light footprint, it is expanded to 24 channels. Of course, other encoding methods such as positional encoding and sinusoidal representation network (SIREN) can also be used, and the present application is not limited thereto.

[0108] Step B2: Decoding the intermediate decoding features, the light direction, and the sight direction through a second decoding unit to obtain rendering reference information.

[0109] It should be understood that since the textures corresponding to different detail levels are usually determined by the light footprint at the shading point, through the above method, a first decoding unit is used to decode the light footprint and material features, so that the decoder can better distinguish and decode the texture features of each detail level, so that the material rendering method provided in the present application can support material rendering at multiple detail levels.

[0110] Reference below Figure 8 , the decoder structure of the above neural network is illustrated with examples. Figure 8 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application. Figure 8As shown, the decoder includes a first decoding unit 801 and a second decoding unit 802. The first decoding unit 801 is a two-layer fully connected neural network. Taking the material feature as a 64-channel input data and the light footprint as a 3-channel input data as an example, after the material feature and the light footprint are input into the first decoding unit 801, the first decoding unit 801 outputs a 64-channel intermediate decoding feature. The second decoding unit 802 is a five-layer fully connected neural network. Taking the sight direction and the light direction as a 4-channel input data as an example, the intermediate decoding feature, the sight direction, and the light direction are input into the second decoding unit 802, and the second decoding unit 802 outputs a 2-channel output data, i.e., the rendering reference information. In addition, the second decoding unit 802 can include residual connections to further enhance the network expression capability. Combined with Figure 8 It can be seen that the first decoding unit and the second decoding unit form a relatively shallow network under the real-time rendering requirement, which can achieve fast operation and thus improve rendering efficiency.

[0111] In addition, the above Figure 8 The decoder structure shown is for illustrative purposes only and does not constitute a limitation on the decoder structure in the present application. Any other network based on machine learning or deep learning and for obtaining rendering reference information based on material features and shading point information can be used as a decoder of the neural network in the embodiments of the present application.

[0112] 504. The computing device renders the material of the shading point based on the rendering reference information.

[0113] In an embodiment of the present application, the rendering reference information includes specular reflection parameters and diffuse reflection parameters of the material to be rendered, that is, parameters of the BRDF, wherein the specular reflection parameters are related to the specular reflection term of the BRDF, and the diffuse reflection parameters are related to the diffuse reflection parameters of the BRDF. It should be understood that there are many ways to implement the BRDF material model, and the material rendering method provided in the present application can be applied to various BRDF material models.

[0114] Schematically, the computing device determines the function value of the bidirectional reflectance distribution function of the shading point based on the rendering reference information, the specular color and the diffuse color of the shading point; based on the function value, the material of the shading point is rendered. For example, the function value is the sum of the diffuse reflection term and the specular reflection term, where the diffuse reflection term = (diffuse reflection parameter × diffuse reflection color) / pi, and the specular reflection term = specular reflection parameter × specular reflection color. It should be understood that this is only an example and does not constitute a limitation of the present application. In actual applications, the function value of the BRDF can be calculated according to needs.

[0115] In addition, during the training process of the neural network provided by the present application, the encoder and the decoder participate in the training together. Since the encoder and the decoder are not trained separately, the obtained encoding and decoding are strongly correlated, so the decoder that needs to be run for each frame does not need to be too deep to have a good decoding effect, which greatly reduces the network operation overhead and ensures the real-time rendering of the material. Schematically, the training process of the neural network can be performed by the aforementioned computing device, or by other servers for training neural networks, and this application does not limit this. Taking the i-th iteration of training the neural network as an example (i is a positive integer), the training process of the neural network is introduced, including the following steps:

[0116] Step C1, inputting sample material information and sample shading point information of the sample shading point into a neural network to obtain predicted rendering reference information of the sample shading point, wherein the predicted rendering reference information includes predicted specular reflection parameters and predicted diffuse reflection parameters of the material to be rendered at the sample shading point.

[0117] The sample material information and sample coloring point information are the same as the material information and coloring point information in the aforementioned step 501, and are not described in detail. In the training process of the neural network, the sample material information and the sample coloring point information can be obtained by randomly sampling real data, for example, and this application does not limit this. In addition, the number of sample material information and sample coloring point information is usually multiple, and the neural network is trained by a large amount of training data, so that the neural network finally trained has good universality and robustness.

[0118] Step C2: determining the loss value based on the predicted rendering reference information and the annotated rendering reference information of the sample shading point, wherein the annotated rendering reference information includes annotated specular reflection parameters and annotated diffuse reflection parameters of the material to be rendered at the sample shading point.

[0119] Among them, a loss function is constructed based on the difference between the predicted rendering reference information and the annotated rendering reference information, and the loss value is calculated based on the loss function. The loss function involved in this application can be various types of loss functions commonly used in the neural network training process, such as absolute value loss function, cosine similarity loss function, square loss function, cross entropy loss function, etc., and this application does not limit this.

[0120] In some embodiments, different weights are used to calculate the loss value for different reflection parameters. Schematically, this step includes: determining a first loss value based on the predicted specular reflection parameters and the marked specular reflection parameters; determining a second loss value based on the predicted diffuse reflection parameters and the marked diffuse reflection parameters; and performing weighted summation of the first loss value and the second loss value to obtain a loss value. In some embodiments, the weight corresponding to the first loss value is greater than the weight corresponding to the second loss value, for example, the weight corresponding to the first loss value is 0.8, and the weight corresponding to the second loss value is 0.2. In this way, since the learning difficulty of the specular reflection parameters is usually greater than that of the diffuse reflection parameters, this method can improve the training effect of the neural network.

[0121] In other embodiments, after the labeled specular reflection parameters and the predicted specular reflection parameters are mapped to the logarithmic space, their loss values ​​are determined. It should be understood that since the distribution of the specular reflection parameters is relatively uneven, this method can change their distribution, reduce the difficulty of learning, and thus improve the training effect of the neural network.

[0122] Step C3: Based on the loss value, adjust the network parameters of the neural network, and perform the i+1th iteration based on the adjusted neural network.

[0123] Wherein, if the loss value or this iteration meets the training end condition, the neural network is output. Wherein, the training end condition is, for example, that the loss value (or error value) is less than the set threshold value. The set threshold value can be set according to actual needs, such as according to the accuracy of the neural network. In other embodiments, the training end condition is that the number of iterations reaches the target number of times, or the training duration reaches the target duration. The present application does not limit the content of the training end condition. In the i-th iteration, if the loss value or this iteration meets the training end condition, indicating that the neural network of this iteration meets the requirements, the neural network model is output, and the trained neural network is obtained. If not satisfied, the network parameters of the neural network are adjusted, and the i+1th iteration is performed based on the adjusted neural network, that is, execution starts again from the aforementioned step C1 until the training is stopped when the training end condition is met to obtain a trained neural network.

[0124] In addition, in some embodiments, during one iteration, multiple sample material information and shading point information are input into the neural network to obtain multiple loss values ​​(or the multiple loss values ​​are processed to obtain a total loss value), and based on the multiple loss values ​​(or the total loss value), it is determined whether the current iteration meets the training end condition. In other implementations, during one iteration, one sample material information and one shading point information are input into the AI ​​model to obtain a loss value, and based on the loss value, it is determined whether the current iteration meets the training end condition. This application does not limit the number of samples in one iteration.

[0125] It should be noted that the training process of the above neural network may also include other steps or other optional implementation methods, which are not limited in this application.

[0126] Reference below Fig. 9 , taking cloth material as an example, the material rendering method in the three-dimensional scene shown in the above steps 501 to 504 is illustrated. Fig. 9 Schematic diagram of a material rendering method in a three-dimensional scene provided by an embodiment of the present application. Fig. 9 As shown, the computing device obtains the material texture map of the cloth material from the scene data of the three-dimensional scene, performs a sampling operation on the material texture map, and obtains the material geometric parameters (warp and weft parameters), that is, the material information of the cloth material. In addition, taking the computing device using the ray tracing mode to render the three-dimensional model as an example, the computing device obtains the shading point information of the shading point from the rendering pipeline, including the light footprint, the light direction, and the line of sight direction. Next, the computing device encodes the material information through the encoder of the neural network network to obtain the material features of the material to be rendered at the shading point, and decodes the material features and the shading point information through the decoder of the neural network to obtain the rendering reference information of the material to be rendered, that is, the specular reflection parameters and the diffuse reflection parameters, and renders the material of the shading point based on the rendering reference information. After that, the computing device can continue to execute other stages in the rendering pipeline.

[0127] In summary, in the material rendering method provided in the embodiment of the present application, after obtaining the material information of the three-dimensional model and the shading point information of the shading point, the material information is extracted and compressed by the encoder of the neural network to obtain the material features of the material to be rendered at the shading point. With the good expression ability of the deep network, the output material features contain rich information and are distinguishable for different material texture maps. Furthermore, the material features and shading point information are decoded by the decoder of the neural network to output the rendering reference information of the material to be rendered, that is, the relevant parameters of the bidirectional reflectance distribution function BRDF. In this way, the calculation process of BRDF is simplified, and the BRDF related parameters can be output quickly and accurately for various materials, so as to achieve high-quality material rendering.

[0128] Moreover, since the network structure of the encoder and decoder is relatively lightweight, the runtime speed can reach the real-time standard; at the same time, the encoder can directly obtain material information in the rendering pipeline, ensuring real-time encoding of material information without prior operation, so the material information can be directly changed during operation, which is convenient to operate.

[0129] In addition, if the encoder implements the encoding process through different encoding units, since the material information of the three-dimensional model does not change in every frame, the encoder can be run for encoding when the material information is updated, and the encoding can be temporarily stored and reused at other times. Moreover, the update of material information may only be related to part of the encoder. For example, the material texture map is not updated, but the material geometric parameters are updated. In this way, there is no need to run the first encoding unit in the encoder, thereby improving the operating efficiency.

[0130] In addition, if the decoder implements the decoding process through different decoding units, since the textures corresponding to different detail levels are usually determined by the light footprint at the shading point, the first decoding unit is used to decode the light footprint and material features in the above manner, so that the decoder can better distinguish and decode the texture features of each detail level, and the final output BRDF related parameters have higher accuracy, and thus the material rendering method provided in the present application can support material rendering at multiple detail levels.

[0131] Fig.10 1 is a schematic diagram of the structure of a material rendering device in a three-dimensional scene provided by an embodiment of the present application. The device can be implemented as part or all of the functions of the aforementioned computing device through software, hardware, or a combination of both. Fig.10 As shown, the device includes an acquisition module 1001, an encoding module 1002, a decoding module 1003 and a rendering module 1004.

[0132] An acquisition module 1001 is used to acquire material information of a 3D model in a 3D scene and shading point information of shading points on the 3D model, wherein the material information indicates a material texture map and material geometric parameters of the 3D model, and the shading point information indicates a lighting condition of the shading point;

[0133] The encoding module 1002 is used to encode the material information through the encoder of the neural network to obtain the material features of the material to be rendered at the shading point, where the material features are used to characterize the texture features and geometric features of the material to be rendered;

[0134] A decoding module 1003 is used to decode the material features and the shading point information through a decoder of a neural network to obtain rendering reference information of the material to be rendered, where the rendering reference information indicates the specular reflection parameters and the diffuse reflection parameters of the material to be rendered;

[0135] The rendering module 1004 is used to render the material of the shading point based on the rendering reference information.

[0136] In some embodiments, the encoder includes a first encoding unit and a second encoding unit; the encoding module 1002 is used to:

[0137] The material texture map is encoded by a first encoding unit to obtain an intermediate material feature;

[0138] The intermediate material features and material geometric parameters are encoded by the second encoding unit to obtain material features.

[0139] In some embodiments, the apparatus further comprises a storage module for at least one of the following:

[0140] Storing intermediate material features to participate in the material rendering process of other shading points when the material texture map is not updated;

[0141] Stores material characteristics so that they can be included in the material rendering process at other shading points if the material information is not updated.

[0142] In some embodiments, the decoder includes a first decoding unit and a second decoding unit, the shading point information indicates the light footprint, light direction and line of sight direction of the shading point, and the light footprint indicates the light distribution on the surface of the shading point; the decoding module 1003 is used to:

[0143] Decoding the light footprint and the material feature through a first decoding unit to obtain an intermediate decoding feature;

[0144] The intermediate decoding features, the light direction, and the sight direction are decoded by the second decoding unit to obtain rendering reference information.

[0145] In some embodiments, the apparatus further comprises a light footprint encoding module, configured to:

[0146] Encoding the light footprint to obtain an encoded light footprint, so as to expand the data length of the light footprint;

[0147] The decoding module 1003 is used to decode the encoded light footprint and material features through the first decoding unit to obtain intermediate decoding features.

[0148] In some embodiments, the rendering module 1004 is used to:

[0149] Determine a function value of a bidirectional reflectance distribution function of the shading point based on the rendering reference information, the specular reflection color and the diffuse reflection color of the shading point;

[0150] Based on the function value, the material of the shading point is rendered.

[0151] In some embodiments, the acquisition module 1001 is used to:

[0152] Get material information and shading point information from the rendering pipeline of the 3D model.

[0153] In some embodiments, the material to be rendered is a cloth material;

[0154] Material texture maps include normal maps and direction maps;

[0155] Material geometry parameters include warp yarn parameters and weft yarn parameters.

[0156] In some embodiments, the training process of the neural network includes:

[0157] In the i-th iteration of training the neural network, the sample material information and the sample shading point information of the sample shading point are input into the neural network to obtain the predicted rendering reference information of the sample shading point, the predicted rendering reference information including the predicted specular reflection parameter and the predicted diffuse reflection parameter of the material to be rendered at the sample shading point, where i is a positive integer;

[0158] Determine the loss value based on the predicted rendering reference information and the annotated rendering reference information of the sample shading point, wherein the annotated rendering reference information includes an annotated specular reflection parameter and an annotated diffuse reflection parameter of the material to be rendered at the sample shading point;

[0159] Based on the loss value, the network parameters of the neural network are adjusted, and based on the adjusted neural network, the i+1th iteration is performed.

[0160] In some embodiments, determining the loss value based on the predicted rendering reference information and the labeled rendering reference information of the sample shading point includes:

[0161] Determining a first loss value based on the predicted specular reflection parameter and the marked specular reflection parameter;

[0162] Determining a second loss value based on the predicted diffuse reflection parameter and the marked diffuse reflection parameter;

[0163] The first loss value and the second loss value are weightedly summed to obtain the loss value.

[0164] Through the above device, after obtaining the material information of the three-dimensional model and the shading point information of the shading point, the material information is extracted and compressed through the encoder of the neural network to obtain the material features of the material to be rendered on the shading point. With the good expression ability of the deep network, the output material features contain rich information and are distinguishable for different material texture maps. Further, through the decoder of the neural network, the material features and shading point information are decoded, and the rendering reference information of the material to be rendered, that is, the relevant parameters of the bidirectional reflectance distribution function BRDF, is output. In this way, the calculation process of BRDF is simplified, and the BRDF related parameters can be output quickly and accurately for various materials, so that high-quality material rendering can be achieved.

[0165] In addition, in the above device, the acquisition module 1001, the encoding module 1002, the decoding module 1003 and the rendering module 1004 can be implemented by software or by hardware. Exemplarily, the implementation of the acquisition module 1001 is described below by taking the acquisition module 1001 as an example. Similarly, the implementation of other modules can refer to the implementation of the acquisition module 1001.

[0166] As an example of a software functional unit, the acquisition module 1001 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 1001 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0167] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0168] As an example of a hardware functional unit, the acquisition module 1001 may include at least one computing device. Alternatively, the acquisition module 1001 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0169] The multiple computing devices included in the acquisition module 1001 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 1001 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 1001 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0170] It should be noted that: the material rendering device in the three-dimensional scene provided in the above embodiment only uses the division of the above functional modules as an example when performing material rendering in the three-dimensional scene. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the material rendering device in the three-dimensional scene provided in the above embodiment and the material rendering method embodiment in the three-dimensional scene belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0171] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first coding unit can be referred to as the second coding unit, and similarly, the second coding unit can be referred to as the first coding unit. The first coding unit and the second coding unit can both be coding units, and in some cases, can be separate and different coding units.

[0172] In this application, the term "at least one" means one or more, and the term "multiple" in this application means two or more, for example, multiple coding units refer to two or more coding units.

[0173] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0174] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0175] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0176] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A material rendering method in a three-dimensional scene, characterized in that: The method comprises: Acquire material information of a three-dimensional model in the three-dimensional scene and shading point information of a shading point on the three-dimensional model, wherein the material information indicates a material texture map and material geometric parameters of the three-dimensional model, and the shading point information indicates a lighting condition of the shading point; The material information is encoded by an encoder of a neural network to obtain material features of the material to be rendered at the shading point, wherein the material features are used to characterize texture features and geometric features of the material to be rendered; Decoding the material feature and the shading point information through a decoder of the neural network to obtain rendering reference information of the material to be rendered, wherein the rendering reference information indicates a specular reflection parameter and a diffuse reflection parameter of the material to be rendered; Rendering the material of the shading point based on the rendering reference information.

2. The method according to claim 1, characterized in that The encoder includes a first encoding unit and a second encoding unit; the encoder through the neural network encodes the material information to obtain the material characteristics of the material to be rendered at the shading point, including: By means of the first encoding unit, the material texture map is encoded to obtain an intermediate material feature; The intermediate material features and the material geometric parameters are encoded by the second encoding unit to obtain the material features.

3. The method according to claim 2, characterized in that The method further comprises at least one of the following: Storing the intermediate material features so as to include the intermediate material features in a material rendering process of other shading points when the material texture map is not updated; The material feature is stored so as to participate the material feature in the material rendering process of other shading points when the material information is not updated.

4. The method according to any one of claims 1 to 3, characterized in that The decoder includes a first decoding unit and a second decoding unit, the shading point information indicates a light footprint, a light direction and a sight direction of the shading point, and the light footprint indicates a light distribution condition on a surface of the shading point; the decoder through the neural network decodes the material feature and the shading point information to obtain rendering reference information of the material to be rendered, including: Decoding the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature; The intermediate decoding feature, the light direction, and the sight direction are decoded by the second decoding unit to obtain the rendering reference information.

5. The method according to claim 4, characterized in that The method further comprises: Encoding the light footprint to obtain the encoded light footprint to expand the data length of the light footprint; The decoding of the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature includes: The encoded light footprint and the material feature are decoded by the first decoding unit to obtain the intermediate decoded feature.

6. The method according to any one of claims 1 to 5, characterized in that The rendering of the material of the shading point based on the rendering reference information includes: Determine a function value of a bidirectional reflectance distribution function of the shading point based on the rendering reference information, the specular reflection color and the diffuse reflection color of the shading point; Based on the function value, the material of the shading point is rendered.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of material information of the three-dimensional model in the three-dimensional scene and shading point information of shading points on the three-dimensional model includes: The material information and the shading point information are obtained from a rendering pipeline of the three-dimensional model.

8. The method according to any one of claims 1 to 7, characterized in that The material to be rendered is a cloth material; The material texture map includes a normal map and a direction map; The material geometric parameters include warp yarn parameters and weft yarn parameters.

9. The method according to any one of claims 1 to 8, characterized in that The training process of the neural network includes: In an i-th iteration of training the neural network, sample material information and sample shading point information of a sample shading point are input into the neural network to obtain predicted rendering reference information of the sample shading point, wherein the predicted rendering reference information includes predicted specular reflection parameters and predicted diffuse reflection parameters of a material to be rendered at the sample shading point, where i is a positive integer; Determine a loss value based on the predicted rendering reference information and the annotated rendering reference information of the sample shading point, wherein the annotated rendering reference information includes an annotated specular reflection parameter and an annotated diffuse reflection parameter of a material to be rendered at the sample shading point; Based on the loss value, the network parameters of the neural network are adjusted, and based on the adjusted neural network, an i+1th iteration is performed.

10. The method according to claim 9, characterized in that The determining of the loss value based on the predicted rendering reference information and the labeled rendering reference information of the sample shading point includes: Determining a first loss value based on the predicted specular reflection parameter and the annotated specular reflection parameter; Determining a second loss value based on the predicted diffuse reflection parameter and the marked diffuse reflection parameter; A weighted sum is performed on the first loss value and the second loss value to obtain the loss value.

11. A material rendering device in a three-dimensional scene, characterized in that: The device comprises: An acquisition module, used to acquire material information of a three-dimensional model in the three-dimensional scene and shading point information of a shading point on the three-dimensional model, wherein the material information indicates a material texture map and material geometric parameters of the three-dimensional model, and the shading point information indicates a lighting condition of the shading point; An encoding module, used for encoding the material information through an encoder of a neural network to obtain material features of the material to be rendered at the shading point, wherein the material features are used to characterize texture features and geometric features of the material to be rendered; A decoding module, configured to decode the material feature and the shading point information through a decoder of the neural network to obtain rendering reference information of the material to be rendered, wherein the rendering reference information indicates a specular reflection parameter and a diffuse reflection parameter of the material to be rendered; A rendering module is used to render the material of the shading point based on the rendering reference information.

12. The device according to claim 11, characterized in that The encoder comprises a first encoding unit and a second encoding unit; the encoding module is used to: By means of the first encoding unit, the material texture map is encoded to obtain an intermediate material feature; The intermediate material features and the material geometric parameters are encoded by the second encoding unit to obtain the material features.

13. The device according to claim 12, characterized in that The device also includes a storage module for at least one of the following: Storing the intermediate material features so as to include the intermediate material features in a material rendering process of other shading points when the material texture map is not updated; The material feature is stored so as to participate the material feature in the material rendering process of other shading points when the material information is not updated.

14. The device according to any one of claims 11 to 13, characterized in that The decoder comprises a first decoding unit and a second decoding unit, wherein the shading point information indicates a light footprint, a light direction and a sight direction of the shading point, and the light footprint indicates a light distribution condition on a surface of the shading point; and the decoding module is used to: Decoding the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature; The intermediate decoding feature, the light direction, and the sight direction are decoded by the second decoding unit to obtain the rendering reference information.

15. The device according to claim 14, characterized in that The apparatus further comprises a light footprint encoding module, configured to: Encoding the light footprint to obtain the encoded light footprint to expand the data length of the light footprint; The decoding of the light footprint and the material feature by the first decoding unit to obtain an intermediate decoding feature includes: The encoded light footprint and the material feature are decoded by the first decoding unit to obtain the intermediate decoded feature.

16. The device according to any one of claims 11 to 15, characterized in that The rendering module is used to: Determine a function value of a bidirectional reflectance distribution function of the shading point based on the rendering reference information, the specular reflection color and the diffuse reflection color of the shading point; Based on the function value, the material of the shading point is rendered.

17. The device according to any one of claims 11 to 16, characterized in that The acquisition module is used to: The material information and the shading point information are obtained from a rendering pipeline of the three-dimensional model.

18. The device according to any one of claims 11 to 17, characterized in that The material to be rendered is a cloth material; The material texture map includes a normal map and a direction map; The material geometric parameters include warp yarn parameters and weft yarn parameters.

19. The device according to any one of claims 11 to 18, characterized in that The training process of the neural network includes: In an i-th iteration of training the neural network, sample material information and sample shading point information of a sample shading point are input into the neural network to obtain predicted rendering reference information of the sample shading point, wherein the predicted rendering reference information includes predicted specular reflection parameters and predicted diffuse reflection parameters of a material to be rendered at the sample shading point, where i is a positive integer; Determine a loss value based on the predicted rendering reference information and the annotated rendering reference information of the sample shading point, wherein the annotated rendering reference information includes an annotated specular reflection parameter and an annotated diffuse reflection parameter of a material to be rendered at the sample shading point; Based on the loss value, the network parameters of the neural network are adjusted, and based on the adjusted neural network, an i+1th iteration is performed.

20. The device according to claim 19, characterized in that The determining of the loss value based on the predicted rendering reference information and the labeled rendering reference information of the sample shading point includes: Determining a first loss value based on the predicted specular reflection parameter and the annotated specular reflection parameter; Determining a second loss value based on the predicted diffuse reflection parameter and the marked diffuse reflection parameter; A weighted sum is performed on the first loss value and the second loss value to obtain the loss value.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images