Material conversion method of three-dimensional model, three-dimensional model rendering method, device and storage medium

Abstract

Embodiments of the present application provide a three-dimensional model material conversion method and a three-dimensional model rendering method, equipment and a storage medium, relating to the technical field of computer vision. The method is based on a differentiable rendering to realize automatic conversion of material parameters of an original model generated by professional modeling software to target material parameters suitable for real-time three-dimensional rendering technology: starting from the final rendering effect, algorithm fitting is performed to approximate the rendering result of the real-time three-dimensional rendering technology to the rendering effect of the professional modeling software, and the target material parameters are gradually obtained in the algorithm fitting process, so that the rendering result in the professional modeling software can be largely restored when rendering is performed through the real-time three-dimensional rendering technology and the target material parameters. The method can convert original material parameters output by various professional modeling software to target material parameters suitable for real-time three-dimensional rendering technology, and the overall time and labor cost is greatly lower than that of a traditional manual scheme, and the overall effect is greatly better than that of other automatic conversion schemes.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for material conversion of a 3D model, a method, device and storage medium for rendering a 3D model. Background Technology

[0002] With the development of Web3D (Web 3-Dimensional) technology, it is now possible to directly present 3D models of objects to be displayed on web pages or applications, and allow users to perform real-time interactions such as rotation and scaling of the 3D model. Leveraging its advantages such as real-time interaction, immediate usability, lightweight deployment, and cross-platform compatibility, Web3D technology has been widely used in e-commerce, immersive marketing, data visualization, and other fields.

[0003] In practical applications, 3D models of objects are typically created using professional modeling software such as 3ds Max and Maya. The raw model files generated by these software programs not only contain complex proprietary geometries, but their material data also often heavily relies on specific rendering engines, making them unsuitable for direct loading and real-time rendering by web applications or other terminal environments. To achieve seamless cross-platform interaction, these raw models generated by professional modeling software need to be converted into a lightweight format suitable for real-time rendering and interactive operations in Web3D technology, such as gITF (Graphics Language Transmission Format).

[0004] Current automated conversion solutions typically employ fixed-rule parametric mapping or static baking techniques. Their core principle is to approximate the complex proprietary material properties of the original model one-to-one with standard PBR (Physically Based Rendering) parameters in lightweight formats (such as glTF) using preset formulas or scripts. However, current automated conversion solutions suffer from insufficient cross-platform fidelity, resulting in significant differences between the real-time visual effects rendered on the terminal based on the converted lightweight format data and the expected effects in professional software, impacting user experience. Summary of the Invention

[0005] This application provides a method for material conversion of a 3D model, a method, device, and storage medium for rendering a 3D model, which can improve the material fidelity of the original 3D model generated by professional modeling software after it is converted to a lightweight format.

[0006] This application provides a method for material conversion of a 3D model, including: acquiring a model file of a target 3D model and multiple sets of scene parameters; parsing geometric data and first material parameters of the target 3D model from the model file using a first renderer; performing forward rendering using the first renderer based on the geometric data, the first material parameters, and the multiple sets of scene parameters to obtain multiple reference images of the target 3D model corresponding to the multiple sets of scene parameters; initializing second material parameters used by a second renderer, wherein the rendering path of the second renderer is adapted to the rendering path of the target application; performing differentiable rendering using the second renderer based on the geometric data, the second material parameters, and the multiple sets of scene parameters to obtain multiple rendered images of the target 3D model corresponding to the multiple sets of scene parameters; updating the second material parameters based on the image reconstruction loss between the multiple rendered images and the multiple reference images, and performing differentiable rendering again based on the updated second material parameters until the image reconstruction loss meets a preset convergence condition to obtain the target material parameters; wherein the target material parameters are used to render the target 3D model in the target application in conjunction with the geometric data.

[0007] This application also provides a three-dimensional model rendering method, including: acquiring a model file of a target three-dimensional model and multiple sets of scene parameters; parsing the geometric data of the target three-dimensional model and the first material parameters under a first renderer from the model file; performing forward rendering using the first renderer based on the geometric data, the first material parameters, and the multiple sets of scene parameters to obtain multiple reference images corresponding to the target three-dimensional model under the multiple sets of scene parameters; initializing the second material parameters used by a second renderer, wherein the rendering path of the second renderer is adapted to the rendering path of the target application; performing differentiable rendering using the second renderer based on the geometric data, the second material parameters, and the multiple sets of scene parameters to obtain multiple rendered images corresponding to the target three-dimensional model under the multiple sets of scene parameters; updating the second material parameters based on the image reconstruction loss between the multiple rendered images and the multiple reference images, and re-performing differentiable rendering based on the updated second material parameters until the image reconstruction loss meets a preset convergence condition to obtain the target material parameters; and rendering the target three-dimensional model in the target application in conjunction with the geometric data based on the target material parameters.

[0008] This application embodiment also provides a three-dimensional model rendering method, including: obtaining target material parameters, the target material parameters being obtained based on the material conversion method of the three-dimensional model described above; and rendering the target three-dimensional model in a target application in combination with geometric data based on the target material parameters, wherein the rendering path of the second renderer is adapted to the rendering path of the target application.

[0009] This application embodiment also provides a computing device, including: a memory and a processor; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor executes the above-mentioned material conversion method or 3D model rendering method for the 3D model.

[0010] This application also provides a computer-readable storage medium storing executable code. When the executable code is executed by a processor of a computing device, the processor executes the aforementioned material conversion method or 3D model rendering method for the 3D model.

[0011] This application also provides a computer program product, including: a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps in the above-described material conversion method or three-dimensional model rendering method for three-dimensional models.

[0012] In this embodiment, a model file of the target 3D model and multiple sets of scene parameters can be obtained. Geometric data and first material parameters under a first renderer are parsed from the model file. Based on this, the first renderer performs forward rendering on the geometric data and first material parameters under multiple sets of scene parameters to obtain multiple reference images. A second renderer performs differentiable rendering on the geometric data and second material parameters under the same multiple sets of scene parameters to obtain multiple rendered images. Using the multiple reference images as monitoring signals, the total loss value between the multiple rendered images and the multiple reference images is calculated. The gradient of the loss value is backpropagated to the second material parameters through the gradient backpropagation mechanism of differentiable rendering, iteratively updating the second material parameters to obtain the target material parameters. Based on this scheme, the second renderer performs material parameter conversion on the target 3D model in a differentiable rendering manner, converting the first material parameters in the target 3D model into target material parameters usable by the target application. Because the rendering path of the second renderer is adapted to the rendering path of the target application, the visual effect presented by real-time rendering based on the converted target material parameters can maintain a high degree of consistency with the expected effect in professional software, solve the problem of inconsistent material performance between different renderers, improve the compatibility and rendering efficiency of 3D models in cross-platform applications, and enhance the user experience. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a material conversion method for a three-dimensional model provided as an exemplary embodiment of this application; Figure 2A flowchart illustrating another method for material conversion of a three-dimensional model provided as an exemplary embodiment of this application; Figure 3 A schematic diagram illustrating the sampling principle for constructing scene parameters, provided as an exemplary embodiment of this application; Figure 4 A schematic diagram illustrating the principle of mapping geometric patches to pixels, provided for an exemplary embodiment of this application; Figure 5 A schematic diagram illustrating the calculation principle of a diffuse reflection hemispherical integral, provided as an exemplary embodiment of this application; Figure 6 A schematic diagram illustrating the process of a second renderer generating a rendered image, provided as an exemplary embodiment of this application; Figure 7 A comparison diagram of model rendering effects provided for an exemplary embodiment of this application; Figure 8 A cumulative distribution function curve for comparing rendering effects is provided as an exemplary embodiment of this application; Figure 9 A schematic diagram of the structure of a material conversion device for a three-dimensional model provided in an exemplary embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computing device provided for an exemplary embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0016] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.

[0017] With the development of Web3D technology, it is now possible to directly present 3D models of objects on web pages or applications, allowing users to interact with these models in real time, such as rotating and scaling. Leveraging its advantages of real-time interaction, immediate usability, lightweight deployment, and cross-platform compatibility, Web3D technology has been widely applied in e-commerce, immersive marketing, and data visualization. For example, in the home furnishing and decoration shopping guide scenario on e-commerce platforms, users can be provided with a real-time preview of product models placed in a showroom or a photo of their own home, along with product layout and matching functions to help users determine if these products meet their shopping needs.

[0018] In practical applications, 3D models of objects are typically created using professional modeling software such as 3ds Max and Maya. The raw model files generated by these software programs not only contain complex proprietary geometries, but their material data also often heavily relies on specific rendering engines, making them unsuitable for direct loading and real-time rendering by web-based or application-based environments. To achieve seamless cross-platform interaction, these raw models generated by professional modeling software need to be converted into a lightweight format suitable for real-time rendering and interactive operations in Web3D technology, such as gITF.

[0019] Manually remodeling would be extremely time-consuming and labor-intensive. Current automated conversion solutions typically employ fixed-rule parametric mapping or static baking techniques. Their core principle is to approximate the complex proprietary material properties of the original model to standard PBR parameters in a lightweight format (such as glTF) using preset formulas or scripts. However, current automated conversion solutions suffer from insufficient cross-platform fidelity, resulting in significant differences between the real-time rendering visuals on the terminal based on the converted lightweight format data and the expected effects in professional software, impacting user experience.

[0020] In view of this, this solution proposes a material conversion method for 3D models. This method, based on differentiable rendering technology, automates the conversion of material parameters of the original model generated by professional modeling software to target material parameters suitable for real-time 3D rendering technology. Starting from the final rendering effect, an algorithm is used to fit the rendering result of real-time 3D rendering technology to approximate the rendering effect of professional modeling software. During the algorithm fitting process, the target material parameters are gradually obtained. Therefore, when rendering on a webpage or in an application using real-time 3D rendering technology and the target material parameters, the rendering result in professional modeling software can be reproduced to a large extent. The method provided in this application can convert the original material parameters output by various professional modeling software into target material parameters suitable for real-time 3D rendering technology. The overall time and labor costs are significantly lower than traditional manual solutions, and the overall effect is significantly better than other automated conversion solutions.

[0021] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 A flowchart illustrating a material conversion method for a three-dimensional model provided in an exemplary embodiment of this application is shown below. Figure 1 As shown, the method includes at least steps 11 to 15 below.

[0023] Step 11: Obtain the model file of the target 3D model and multiple sets of scene parameters, and parse the geometric data of the target 3D model and the first material parameters under the first renderer from the model file.

[0024] Step 12: Based on the geometric data, the first material parameters, and multiple sets of scene parameters, perform forward rendering using the first renderer to obtain multiple reference images of the target 3D model under multiple sets of scene parameters.

[0025] Step 13: Initialize the second material parameters used by the second renderer, and adapt the rendering path of the second renderer to the rendering path of the target application.

[0026] Step 14: Based on the geometric data, the second material parameters, and multiple sets of scene parameters, use the second renderer to perform differentiable rendering to obtain multiple rendered images of the target 3D model under multiple sets of scene parameters.

[0027] Step 15: Based on the image reconstruction loss between multiple rendered images and multiple reference images, update the second material parameters, and re-perform differentiable rendering based on the updated second material parameters until the image reconstruction loss meets the preset convergence condition to obtain the target material parameters; wherein, the target material parameters are used to render the target 3D model in the target application in combination with geometric data.

[0028] In this embodiment, the target 3D model is a 3D model of the target object. The target object can be any object, such as an animal, plant, building, person, etc., or various goods that can be sold or displayed on an e-commerce platform, such as vehicles, mobile phones, tables, sofas, clothes, sporting goods, etc. This embodiment does not impose any restrictions. In an e-commerce scenario, the target object can be a product with a 3D model on the e-commerce platform, such as an air conditioner, television, or wardrobe, etc. This embodiment does not impose any restrictions.

[0029] The target 3D model is generated using professional modeling software, which embeds a first renderer. The material system used in conjunction with the first renderer is called the first material system. For example, the professional modeling software can be 3ds Max, Maya, Blender, or other 3D modeling software; the first renderer can be a V-Ray renderer; and the first material system can be a V-Ray material system. This application does not impose any limitations on this.

[0030] The model file of the target 3D model includes geometric data and primary material parameters. The geometric data is used to construct the 3D spatial shape and surface topology of the target 3D model, and may include information such as vertex coordinates, face indices, and texture coordinates. In the rendering process, material refers to the combination of multiple visual attributes of an object's surface. The primary material parameters define the visual attributes of the target 3D model's surface, such as diffuse color, reflectivity, gloss, and refractive index. The primary material parameters, the secondary material parameters described below, and the target material parameters described below are numerical arrays, all of which can be implemented based on textures. That is, specific material properties at different locations on the model's surface are defined using one or more image files. For example, material parameters include diffuse maps, metallic maps, and roughness maps.

[0031] The first renderer performs forward rendering in an offline environment, meaning it calculates the final 2D image based on known geometric data, material parameters, and scene parameters using algorithms such as ray tracing. Forward rendering means that the input parameters for the rendering process are known and deterministic, and the output 2D image is obtained through one-way calculation using a physically based lighting model, without needing to deduce the input parameters from the image. By rendering in this way, the first renderer can simulate the lighting environment of the real world, presenting realistic visual effects through the interaction of primary material parameters and light. This offline rendering method can generate high-precision rendering results, but the computation time for each frame is relatively long, making it difficult to meet the needs of real-time interaction.

[0032] The target application uses a renderer employing real-time 3D rendering technology. It utilizes lightweight data for real-time rendering, completing lighting calculations, material shading, and image output for the 3D scene in a very short time, while also supporting user interaction. Real-time 3D rendering technology can be either Web3D or native mobile rendering technology; there are no restrictions. For example, Web3D technology can use glTF as the standard transmission format for 3D models and PBR materials as the standard material system. PBR is a rendering technology that simulates the realistic physical interaction between light and the surface of an object. Following the law of conservation of energy and optical principles, it achieves realistic appearance effects under different lighting conditions through parameters such as reflectivity, roughness, and metallicity, along with corresponding textures, making 3D models more natural and realistic. PBR materials are a physically-compliant material description method that defines the visual properties of the model surface through parameters such as metallicity maps and roughness maps, making them suitable for use in real-time rendering environments.

[0033] Since the first renderer differs from the renderer used by the target application, in order to adapt the high-quality 3D model generated by professional modeling software to real-time rendering environments such as terminal applications or web pages, in this embodiment, a second renderer is used to perform material parameter conversion on the target 3D model in a differentiable rendering manner, so as to convert the first material parameters in the target 3D model into target material parameters that can be used by the target application. Here, the rendering path of the second renderer is adapted to the rendering path of the target application. Adaptation means that the second renderer and the target application are the same or similar in terms of rendering effect or rendering logic, so that the rendered image can reflect the actual display effect in the target application. On the one hand, the rendered image rendered by the second renderer has the same or similar visual effect as the rendered image generated by the target application in actual operation. Under the same input, the output images of the two are highly consistent in visual features such as color, lighting, shadow, and reflection, or have similar visual perception. On the other hand, the second renderer is configured with rendering logic and rendering parameters corresponding to the target application, including but not limited to lighting models, shader calculation methods, texture sampling settings, and rendering pipelines. By implementing the same, similar, or equivalent configurations at the rendering logic level, the second renderer can simulate the real rendering behavior of the target application during the calculation process, thereby ensuring that the rendered image it outputs can accurately reflect the actual display effect in the target application, and providing a reliable supervision signal for subsequent material parameter optimization based on differentiable rendering.

[0034] In practical applications, the target material parameters are rendered within the target application, enabling 3D digital product displays in e-commerce (such as 3D home decoration and furnishing guides), immersive virtual showrooms, and interactive scenarios like online virtual try-on. This ensures the model displays accurate light and shadow reflections and material details under any dynamic lighting environment, significantly lowering the barrier to asset creation while dramatically improving the user's visual experience. Optionally, the target application can be a browser, application, or mini-program on electronic devices such as laptops, mobile phones, tablets, and smart interactive devices. Users can perform interactive operations such as rotating, scaling, and moving the 3D model within the target application, obtaining a smooth real-time rendering experience.

[0035] Differentiable rendering is a technique that makes the rendering process in traditional computer graphics differentiable. It allows the calculation of the gradient of the rendered 2D image relative to input parameters (such as geometry, material parameters, lighting parameters, camera viewpoint, etc.), thereby embedding the rendering process into a machine learning optimization framework. By adjusting the input parameters through backpropagation, it achieves inverse rendering of the scene from the image. In this embodiment, differentiable rendering automatically converts the first material parameter into the target material parameter, eliminating the need for manual material mapping rules. This efficiently completes the cross-platform conversion of material parameters from offline renderers to real-time renderers while maintaining the realism of the rendered image.

[0036] In specific implementations, such as Figure 2 As shown, multiple scenes can be pre-constructed using a scene construction system, each scene including a set of scene parameters. The model file of the target 3D model is parsed using a parsing system to obtain the geometric data and first material parameters of the target 3D model. For any set of scene parameters, on the one hand, under any set of scene parameters, the first renderer is used to perform forward rendering on the geometric data bound to the first material parameters to obtain a reference image of the target 3D model under the scene parameters. The reference image is a high-fidelity image obtained based on ray interaction calculations using the first material parameters, serving as the ground truth for subsequent parameter optimization.

[0037] In one exemplary embodiment, the aforementioned "using a first renderer to perform forward rendering based on geometric data, first material parameters, and multiple sets of scene parameters to obtain multiple reference images of the target 3D model corresponding to multiple sets of scene parameters" can be implemented as follows: For any set of scene parameters, a target format rendering description file is generated based on the first material parameters, geometric data, and any set of scene parameters; the rendering description file is input into the first renderer, and a target 3D model is constructed based on the geometric data in the rendering description file. The target 3D model includes multiple geometric faces and their corresponding vertices; based on the camera pose and model pose in the rendering description file, the multiple geometric faces are projected onto the imaging plane and rasterized to obtain a second pixel point on the imaging plane covered by multiple geometric faces; based on the first material parameters and lighting conditions in the rendering description file, the second pixel point is shading calculated to obtain a reference image of the target 3D model under any set of scene parameters.

[0038] The scene parameters, geometric data, and first material parameters obtained through the scene construction and parsing systems may differ in format from those of the first renderer. Through format conversion and encapsulation, a target format rendering description file conforming to the first renderer's interface specification is generated. During this process, the first material parameters are automatically evaluated and adjusted based on the scene parameters, such as fine-tuning diffuse brightness in low-light environments or correcting specular properties according to the ambient color temperature, to ensure material blending with the scene. After reading the rendering description file, the first renderer reconstructs the target 3D model in virtual 3D space based on the geometric data within. Next, based on the camera pose and model pose defined in the rendering description file, the first renderer projects multiple geometric patches in 3D space onto a 2D imaging plane through transformation operations of the view matrix and projection matrix. Subsequently, visibility determination and sampling processing are performed to identify second pixels on the imaging plane covered by multiple geometric patches, and the depth value and patch index corresponding to each second pixel are recorded, thus completing the mapping from vector geometry to discrete pixels. For each covered second pixel, the first renderer combines the first material parameters and lighting conditions from the rendering description file, and calls its built-in physically based shading engine to perform ray interaction calculations to obtain the final color value (RGB) of the second pixel. The color values ​​of all second pixels are then combined to obtain a reference image of the target 3D model under a set of scene parameters.

[0039] On the other hand, the second material parameters used by the second renderer are initialized. The initial texture of the second material parameters can be a solid color, random noise, or an arbitrary placeholder image, or the value of the first material parameters can be used; each pixel in the texture is treated as a free variable to be optimized. The scene parameters are converted into a data format, and the converted scene parameters, geometric data, and second material parameters are input into the second renderer as differentiable rendering parameters. The second renderer performs differentiable rendering of the geometric data based on the currently predicted second material parameters and scene parameters to obtain the rendered image of the target 3D model under the scene parameters.

[0040] For any set of scene parameters, the rendered image is compared with the corresponding reference image to calculate the loss value between the rendered image and the reference image. A total loss value is constructed based on multiple loss values ​​corresponding to multiple sets of scene parameters. Based on the total loss value, the gradient of the second material parameter is calculated using the backpropagation algorithm. The gradient information precisely indicates the direction and magnitude of the rendering error caused by the current second material parameter (e.g., too high metallicity or too low roughness). Then, the value of the second material parameter is updated based on the gradient. For example, each pixel value in the texture is updated; for instance, the red channel value of a pixel in the diffuse texture is decreased, and a pixel in the metallic texture is brightened.

[0041] The material parameters are jointly trained under multiple sets of scene parameters. The rendering, comparison, and update steps described above are repeated until the total loss value meets the preset convergence condition (e.g., the total loss value is less than a preset threshold or the number of iterations reaches the maximum value). This yields target material parameters that exhibit good rendering effects under multiple sets of scene parameters. The target material parameters and the second material parameters belong to the same material system; for example, they can be PBR material parameters. The calculation method for the loss value can include, but is not limited to, pixel-level color difference, structural similarity difference, or perceptual loss.

[0042] In an exemplary embodiment, the scene construction system generates multiple sets of scene parameters, including: sampling a preset parameter space according to parameter sampling rules to generate multiple sets of scene parameters; wherein the preset parameter space includes a lighting condition parameter space, a camera pose parameter space, and a model pose parameter space, and a set of scene parameters includes the corresponding camera pose, model pose, and lighting conditions.

[0043] The preset parameter space is a high-dimensional space, defined by multiple independent parameter dimensions that affect the rendering result. Camera pose refers to the set of parameters defining the virtual camera's position coordinates and orientation angle in 3D space, used to determine the viewing angle of the target 3D model. Lighting conditions define the type, intensity, color, and distribution of light sources in the scene, specifically including ambient light (such as HDR (High Dynamic Range) image lighting) and directional light parameters. Model pose refers to the spatial transformation parameters of the target 3D model in the scene coordinate system, including the model's translation vector, rotation angle, and scaling ratio, used to determine the target 3D model's placement and rotation angle in the scene.

[0044] Optionally, when constructing the scene, multiple sets of scene parameters can be generated by sampling a preset parameter space according to parameter sampling rules. Parameter sampling rules include, but are not limited to, spherical uniform sampling rules, random sampling rules, mesh sampling rules, discrete enumeration rules, etc. This application embodiment does not limit the specific type of parameter sampling rules. For example, multiple sets of scene parameters can be constructed based on spherical sampling rules. For example, the geometric center of the target 3D model is taken as the origin of the sphere's center coordinates. According to a preset spherical sampling algorithm, multiple uniformly distributed sampling points are determined on a virtual sphere centered on the sphere's center. The 3D coordinates of each sampling point represent a camera position, with the camera's line of sight pointing towards the target 3D model at the sphere's center, thus forming a camera pose. Figure 3 The results of uniform sampling on a unit sphere are shown, where the coordinates of each sampling point are represented in the form of (X, Y, Z). It should be noted that... Figure 3 The sampling point distribution shown is for illustrative purposes only; in practical applications, the number of sampling points and the radius of the sphere can be adjusted according to requirements. Additionally, different lighting characteristics (e.g., multiple HDR images with different lighting characteristics) can be loaded from the lighting condition parameter space as lighting conditions to cover various lighting environments, ranging from warm to cool tones and from indoor to outdoor light. Through this method, multiple sets of scene parameters containing different combinations of viewpoints and lighting combinations can be generated.

[0045] In addition to dynamically generating multiple sets of scene parameters through parameter sampling rules, rendering scenes can also be quickly selected from pre-configured scene parameters. For example, preset scenes containing different numbers of viewpoints and different lighting combinations can be pre-set, including but not limited to: 24-view, 16-view, or 8-view panoramic planar scenes, 8-view multi-HDR scenes, and 8-view multi-HDR cube scenes. A panoramic planar scene refers to a scene configuration where the camera surrounds the target 3D model in a horizontal plane. In this scene, the target 3D model is placed at the center of the scene, and the cameras are distributed along a circular trajectory on the horizontal plane. The camera height is fixed, and the line of sight points to the center of the model. Taking an 8-view panoramic planar scene as an example, the cameras are evenly distributed at 45° intervals in the horizontal direction, that is, the camera azimuth angles are 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, respectively. The camera elevation angle is fixed at 0° (i.e., horizontal shooting), and the distance between the camera and the center of the model is a preset value. Each set of camera poses corresponds to one scene parameter, generating a total of 8 sets of scene parameters. Multiple HDR scenes refer to scene configurations that use multiple HDR maps with different lighting characteristics to render a fixed camera pose, used to evaluate the appearance of a target 3D model under different lighting conditions. For example, if the HDR map library contains maps with 3 different lighting characteristics, then an 8-view multiple HDR scene can generate 24 sets of scene parameters for 8 viewing angles and 3 lighting conditions. Multiple HDR cube scenes are an extension of multiple HDR scenes. In a cube scene, the camera pose is no longer limited to the horizontal plane, but is distributed in multiple directions in 3D space, used to more comprehensively evaluate the appearance of a target 3D model in multiple directions in 3D space.

[0046] In this embodiment, the multiple sets of scene parameters serve to construct a high-dimensional constraint space for comprehensive calibration of the target material parameters. By constructing multiple sets of scene parameters encompassing different lighting conditions, model poses, and camera poses, and jointly training the material parameters under these multiple sets of scene parameters, local overfitting during material parameter optimization can be effectively avoided, improving the generalization ability of the target material parameters in different environments. The final target material parameters represent the objective physical properties of the object's surface. Even after leaving the training environment, they still possess the ability to maintain physically correct lighting and shadow interactions in any unknown scene, accurately reproducing the desired material texture during rendering in the target application.

[0047] In an exemplary embodiment, the aforementioned "using a second renderer to perform differentiable rendering based on geometric data, second material parameters, and multiple sets of scene parameters to obtain multiple rendered images of the target 3D model corresponding to multiple sets of scene parameters" can be implemented as follows: For any set of scene parameters, geometric data, second material parameters, and any set of scene parameters are input into the second renderer to construct a target 3D model based on the geometric data in the second renderer. The target 3D model includes multiple geometric faces and their corresponding vertices. Based on the camera pose and model pose in any set of scene parameters, the multiple geometric faces are projected and differentiable rasterized to obtain a first pixel point on the projection plane covered by multiple geometric faces. Based on the texture coordinates of the vertices corresponding to the geometric faces covering the first pixel point, the material properties of the first pixel point are sampled from the second material parameters. Based on the lighting conditions in any set of scene parameters and the material properties of the first pixel point, the color information of the first pixel point is calculated. Based on the first pixel point and its texture coordinates, material properties, and color information, a rendered image of the target 3D model under any set of scene parameters is obtained.

[0048] The second renderer loads geometric data, second material parameters, and any set of scene parameters. The second material parameters, camera pose, model pose, and optional directional lighting information are all configured as parameters and then input into the second renderer, which can use them directly without parsing. The camera pose and model pose can be represented by matrices.

[0049] For geometric data, this embodiment uses a file format for input. In some optional embodiments, OBJ file format can be used for input. OBJ is a plain text format used to store static geometric data. The second renderer calls the mesh loader (Obj Loader) to read the text content of the input OBJ file line by line to parse the OBJ file. The parsed data is extracted as the underlying geometric properties required for rendering. Specifically, the mesh loader parses the vertex data in the OBJ file to obtain the set of point coordinates of the model in three-dimensional space, which is used to define the shape basis of the model. It parses the face indices in the face definition of the OBJ file to determine how the vertices connect to form triangular faces in order to establish the topological structure of the model. It parses the vertex normal data in the OBJ file to obtain the orientation information of the vertices or faces. This information is used in subsequent lighting calculations to determine the light reflection angle and shadow generation.

[0050] For lighting data, in some optional embodiments, an HDR image can be used as the ambient light input. The second renderer calls the ambient light loader (HDRLoader) to process the ambient light map file (e.g., env.hdr), converting the HDR image containing real-world lighting information into a lighting data structure that the renderer can recognize. Specifically, the HDR image file is loaded, and its high dynamic range color information is read. The read data is converted into lighting information for IBL (Image-Based Lighting) as the ambient background light source for the scene. Further optionally, the loaded ambient light map is pre-filtered using a multi-level mipmap generator to generate a multi-level pre-filtered ambient map for subsequent specular calculations.

[0051] After completing the above data loading and preprocessing, the second renderer sequentially executes four processing stages: vertex shader, rasterization, fragment shader, testing, and blending, to render the target 3D model and obtain a rendered image under any set of scene parameters.

[0052] The vertex shader is the first processing stage in the rendering pipeline. Its core function is to transform the 3D model from model space to the projection plane and pass necessary vertex attributes to subsequent stages. Based on the vertex index relationships recorded in the face index array, the vertex shader connects discrete vertices into geometric faces (such as triangle faces), and connects these geometric faces to construct a complete target 3D model. Each geometric face includes multiple vertices and their corresponding normals and texture coordinates.

[0053] After the target 3D model is constructed, the vertex shader projects multiple geometric patches based on the camera pose and model pose in any set of scene parameters. Specifically, this includes: transforming the vertices of multiple geometric patches from model space to camera space based on the camera pose and model pose in any set of scene parameters, and then projecting the transformed vertices onto the projection plane to obtain the projection regions corresponding to each of the multiple geometric patches.

[0054] First, the target 3D model is transformed based on its pose, changing it from the local coordinate system of model space to the world coordinate system. Then, based on the camera pose, the vertices in the world coordinate system are transformed to the coordinate system of the camera space. Finally, through perspective projection transformation, the vertices in the camera coordinate system are mapped onto a 2D projection plane, obtaining the 2D coordinates of each vertex on the projection plane, as well as the projection areas corresponding to multiple geometric patches. Simultaneously, the vertex shader transforms the vertex normal information to view space and passes the texture coordinates as is to the next stage for subsequent material sampling.

[0055] After the projection transformation is completed, the second renderer performs differentiable rasterization on the projection area of ​​each geometric facet to determine whether each pixel on the projection plane is covered by the projection area, and takes the pixel covered by the projection area as the first pixel.

[0056] Rasterization is the second processing stage of the rendering pipeline, used to convert geometric faces into a set of pixels on a projection plane. The rasterizer receives the geometric faces and the coordinates of their vertices in the 2D projection plane from the vertex shader. For the projection region of any geometric face, the rasterizer first calculates its boundary equations: based on the 2D coordinates of multiple vertices of the geometric face on the projection plane, it calculates the equations of the lines corresponding to multiple edges of the geometric face. This boundary equation is used to subsequently determine the positional relationship between pixels and the geometric face. For each pixel on the projection plane, the rasterizer determines whether the center of each pixel is located inside the projection region based on the boundary equations. The determination method is as follows: substitute the 2D coordinates of the pixel center into the boundary equations of multiple edges. If multiple calculation results all satisfy a preset sign condition (e.g., all greater than or equal to zero), then it is determined that the center of the pixel is located inside the projection region of the geometric face, that is, the pixel is covered by the projection region of the geometric face.

[0057] In an exemplary embodiment, after determining the covered pixel, the texture coordinates of the corresponding vertex of the geometric patch covering the first pixel and the centroid coordinates of the first pixel within the geometric patch are obtained; the texture coordinates of the vertex are interpolated using the centroid coordinates to obtain the texture coordinates of the first pixel; and the material properties of the first pixel are obtained by sampling from the second material parameters based on the texture coordinates of the first pixel.

[0058] In the embodiments of this application, the rasterizer is a differentiable rasterizer, such as DIB-R (Differentiable Interpolation-based Renderer). The differentiable rasterizer models the rendering process as a differentiable probabilistic process, providing a complete and differentiable forward path for the gradient propagation of material parameters, so that the gradient can propagate backward from the final rendered image to the 3D geometric vertices and material parameters.

[0059] Regarding the association between pixels and geometric patches, two cases are handled differently: foreground pixels and background pixels. For foreground pixels covered by geometric patches, the rasterizer determines visibility through depth testing. Unlike traditional renderers that use discrete depth comparisons, the differentiable rasterizer softens the depth test into a weighted averaging process: for multiple geometric patches covering the same pixel, contribution weights are calculated based on the depth values ​​of the multiple geometric patches at that pixel, with smaller depth values ​​resulting in larger contribution weights. Then, using the centroid coordinates of the pixel within its assigned geometric patch, the attributes of multiple vertices of the geometric patch (including normals, texture coordinates, etc.) are interpolated to obtain the normal and texture coordinates of that pixel. For example, as shown... Figure 4 As shown, taking a triangular facet as an example, if the texture coordinates of the three vertices are (u0, v0), (u1, v1), and (u2, v2), and the centroid coordinates are (α, β, γ), then the texture coordinates (u, v) of a pixel can be calculated through centroid coordinate interpolation: u = α·u0 + β·u1 + γ·u2, v = α·v0 + β·v1 + γ·v2. Since this interpolation process is differentiable, foreground pixels covered by a specific facet can propagate the gradient back to the corresponding vertex's attributes through the interpolation relationship.

[0060] For background pixels not covered by any triangular facets, the differentiable rasterizer employs a distance-based probabilistic association method. Specifically, for each background pixel, the rasterizer calculates the distance from the background pixel to the projection regions of multiple triangular facets, and calculates a probability value based on the distance; the closer the distance, the higher the probability value, and the farther the distance, the lower the probability value. In this way, a smooth probabilistic association is established between the background pixel and all triangular facets, rather than completely ignoring them. This allows the background pixel to also backpropagate its gradient to the vertices of all triangular facets through the distance function, thus providing an effective gradient supervision signal for the geometric contour during the optimization process.

[0061] After rasterization, for each foreground pixel (i.e., the first pixel), its normal and texture coordinates are passed to the fragment shader stage; for each background pixel, its probability value is passed as weight information for subsequent pixel compositing. The fragment shader is the third processing stage of the rendering pipeline, and its core function is to determine the final color of multiple first pixels. The fragment shader receives the texture coordinates and normals of multiple first pixels passed from the rasterization stage, samples them from the second material parameters based on the texture coordinates, and obtains the material properties of the first pixels.

[0062] In one exemplary embodiment, the second material parameter is stored in the form of multiple two-dimensional texture maps, including a diffuse map, a metallicity map, and a roughness map. The diffuse map, also known as an albedo map, stores the inherent color information of the object's surface. The diffuse map is an RGB three-channel image that records the color and proportion reflected back by the object under pure white light, without including any effects from illumination, shadows, or highlights. For example, the diffuse map of a red plastic ball appears as a uniform red. During optimization, the RGB values ​​of multiple pixels in the diffuse map are adjusted using a backpropagation algorithm, gradually matching the object's color distribution to the appearance in the target image.

[0063] Metallization maps store information about the metallic properties of an object's surface. A metallization map is a single-channel grayscale image with pixel values ​​ranging from 0 to 1. 0 represents non-metallic materials (such as plastic, wood, and stone), 1 represents metallic materials (such as gold, silver, and copper), and values ​​between 0 and 1 represent materials in between (such as rusty metal). Metallization values ​​directly affect the color and intensity of specular highlights: non-metallic highlights are white, while metallic highlights are colored by the diffuse map. During optimization, by adjusting the grayscale values ​​of multiple pixels in the metallization map, the metallic texture of the object (such as the color and intensity of specular highlights in metallic areas) is gradually matched to the material representation in the target image.

[0064] Roughness maps store information about the smoothness of an object's surface. A roughness map is a single-channel grayscale image with pixel values ​​ranging from 0 to 1. 0 represents a perfectly smooth surface (like a mirror), where incident light is concentrated and reflected, producing sharp, clear highlights; 1 represents a perfectly rough surface (like frosted glass), where incident light is scattered in multiple directions, producing diffuse, muted highlights. The roughness value determines the size and sharpness of the highlight area. During optimization, by adjusting the grayscale values ​​of multiple pixels in the roughness map, the smoothness of the object's surface (such as the dispersion or concentration of highlights) is gradually matched to the visual features in the target image.

[0065] The three texture maps mentioned above establish a mapping relationship between texture coordinates and the surface of the 3D model. Based on the texture coordinates of the first pixel, the material properties of the first pixel are obtained by sampling from the second material parameters, including: sampling diffuse color values ​​from the diffuse map, metallic values ​​from the metallic map, and roughness values ​​from the roughness map, using the texture coordinates of the first pixel as sampling coordinates; and combining the diffuse color value, metallic value, and roughness value into the material properties of the first pixel.

[0066] The fragment shader uses texture coordinates (u, v) as sampling coordinates to read the pixel values ​​at corresponding positions in three texture maps. The diffuse map returns the color values ​​of the RGB channels as the diffuse attribute; the metallic and roughness maps return the values ​​of a single channel as the metallic and roughness attributes, respectively. Thus, each first pixel obtains a complete set of material properties, including diffuse color value, metallic value, and roughness value, for subsequent lighting calculations.

[0067] The lighting conditions include at least one directional light and / or ambient light. The second renderer calculates the color of the first pixel based on the lighting conditions in any set of scene parameters and the material properties of the first pixel, including: for any directional light, calculating the directional lighting color generated by the directional light on the first pixel based on the material properties of the first pixel and the lighting direction and intensity of the directional light; for ambient light, calculating the ambient lighting color generated by the ambient light on the first pixel based on the material properties of the first pixel and the radiance contribution of the ambient light in multiple incident directions; and obtaining the color information of the first pixel based on the directional lighting color generated by at least one directional light on the first pixel and / or the ambient lighting color generated by the ambient light on the first pixel.

[0068] Directional light simulates parallel light rays (such as sunlight) emanating from infinity, with its direction remaining constant within the scene. Ambient light simulates scattered light from multiple directions in the surrounding environment (such as skylight, wall reflections, etc.). In this embodiment, both directional light color and ambient light color are calculated using the PBR rendering equation. The PBR rendering equation is based on microplane theory and the law of conservation of energy. Its core BRDF (Bidirectional Reflectance Distribution Function) consists of diffuse and specular terms, expressed as follows: Where c is the albedo value sampled from the diffuse map, and k d It is the proportion of energy diffusely reflected, k s It is the energy ratio of specular reflection, both of which are determined by the metallicity value and satisfy the law of conservation of energy, k d +k s =1. D is the normal distribution function, driven by the roughness value, controlling the normal distribution of the micro-surface and determining whether the material is smooth or rough. F is the Fresnel function, determined by both metallicity and albedo, describing the proportion of light reflected when incident at different angles. G is the geometric occlusion function, driven by the roughness value, describing the degree of mutual occlusion and self-shadowing between micro-facets. n is the surface normal, ω i Let ω be the incident direction. o ω represents the direction of emission (i.e., the direction of observation).i · n is the cosine of the angle between the ray and the normal, ω o ·n is the cosine of the angle between the line of sight and the normal, L i (p,w i ) represents the incident light, dw i It is a differential solid angle.

[0069] The PBR rendering equation describes the rendering of a single pixel p in a given incident direction ω. i and the direction of observation ω o The emitted radiance L o In other words, from the direction of observation ω o Looking at it from the surface, the final color of point p is L. o It is the incident light L from all directions i The sum of contributions. For directional light, since the light source comes from a single incident direction, the hemispherical integral in the above expression degenerates into a single calculation in that direction, directly taking the light source direction as ω. i Substituting these values ​​into the equation yields the directional lighting color. Specifically, for any directional light fragment shader, the first step is to obtain the lighting direction vector ω of that direction. i and light intensity L i The direction of the light source and the direction of observation ω o Substituting the surface normal n, the albedo c, metallicity, and roughness obtained from the material map into the PBR rendering equation above (omitting the integral sign and directly calculating the integrand), we obtain the outgoing radiance L produced by the light in that direction at the pixel. o This refers to the directional light color produced by the directional light at the first pixel. If there are multiple directional lights, the calculation results of the multiple directional lights are accumulated to obtain the directional light color produced by the directional light at the first pixel.

[0070] For ambient light, the ambient light during rendering is calculated from each direction on the hemisphere, requiring hemispherical integration over all incident directions within the hemispherical space. Calculating hemispherical integration in real-time rendering is overly complex. In some alternative implementations, IBL (Integrated Light Booster) technology can be used to accelerate the calculation of the above equations through irradiance mapping, a pre-filtered environment map, and an integral lookup table to obtain the ambient light color. The irradiance map stores the diffuse incident light integration results in multiple normal directions. The pre-filtered environment map is a multi-level asymptotic texture; different levels correspond to specular incident light integration results at different roughnesses. Lower roughness results in clearer layers, while higher roughness results in more blurred layers. The integral lookup table is a two-dimensional lookup table used to correct the contribution of the pre-filtered environment map to specular reflection based on roughness and camera pose.

[0071] In an exemplary embodiment, calculating the ambient light color generated at the first pixel based on the material properties of the first pixel and the radiance contribution of ambient light in multiple incident directions includes: processing the ambient light to obtain an irradiance map, a pre-filtered environment map, and an integral lookup table; determining the albedo based on the diffuse map, where the albedo represents the ability of an object surface to diffusely reflect incident light; determining the base reflectivity based on the metallicity map and the diffuse map, where the base reflectivity represents the proportion of energy directly specularly reflected when light is incident perpendicularly on the object surface; calculating the diffuse color based on the irradiance map and the albedo; calculating the specular color based on the pre-filtered environment map, the integral lookup table, and the base reflectivity; and adding the diffuse color and the specular color to obtain the ambient light color.

[0072] Based on the input HDR image, the diffuse term k in the PBR rendering equation is pre-processed. d The integral of c / π is performed in hemispherical space, storing the integrated results of diffuse incident light along multiple normal directions to obtain an irradiance map. For a given normal direction n, the irradiance map sampled yields... The integral value is obtained by first integrating the specular term in the PBR rendering equation under different roughnesses and storing it as a multi-level mipmap to obtain a pre-filtered environment map. For a given reflection vector and roughness, the sampled value from the pre-filtered environment map is... The integral value is calculated in advance. The portion of the specular integral related to the fundamental reflectivity is pre-calculated and stored as a two-dimensional integral lookup table. The input to the integral lookup table is the roughness and the dot product n·ω of the normal and the viewing direction. o The output is a scaling factor (scale) and an offset (bias).

[0073] In one exemplary embodiment, based on the input HDR ambient light, accelerated computation can be performed during the preprocessing stage using the parallel computing capabilities of a GPU (Graphics Processing Unit) to generate an irradiance map, a pre-filtered environment map, and an integral lookup table. For example, parallel computation can be implemented using CUDA (Compute Unified Device Architecture). Specifically, the irradiance map is obtained by traversing the hemispherical pixels of the ambient light map and accumulating the diffuse reflection integrals corresponding to multiple normal directions; the pre-filtered environment map uses a segmentation and summation approximation algorithm to calculate multi-level asymptotic textures using parallel computing capabilities; and the integral lookup table pre-calculates the portion of the specular integral related to the base reflectivity, thereby significantly improving computational efficiency.

[0074] The diffuse component of ambient lighting is calculated as follows: Diffuse color = Incident irradiance · c / π. The fragment shader samples the incident irradiance in the irradiance map based on the normal direction n of the pixel. Figure 5 The calculation principle of diffuse reflection hemispherical integral is illustrated: with the shading point P as the center, the incident light at multiple spherical sampling positions is integrated and accumulated along the surface normal N direction in the hemispherical space, and the diffuse reflection color at that point is finally obtained. The irradiance map is a pre-calculated and stored representation of this integral result.

[0075] The specular highlight (i.e., specular color) in ambient lighting is calculated as follows: Specular color = Pre-filtered environment map sampled lighting value × (Base reflectivity × Scaling factor + Offset). The fragment shader calculates the reflection vector based on the normal and the view direction, determines the sampling mipmap level based on the pixel's roughness value, and samples the pre-filtered lighting value from the pre-filtered environment map with the reflection vector as the sampling direction. Simultaneously, based on the dot product of the normal and the view direction, and the roughness value, the scaling factor and offset are sampled from the integral lookup table. Then, the base reflectivity is calculated based on the metallic and diffuse maps: Base reflectivity = Diffuse color value × Metallic value + 0.04 × (1 - Metallic value). The diffuse color and specular color are added together to obtain the ambient lighting color generated by the ambient light at the first pixel.

[0076] The fragment shader adds the directional lighting color to the ambient lighting color to obtain the final color information of the first pixel. If both directional and ambient lighting exist in the scene, the final color of the first pixel is the sum of the two. If only directional lighting exists, the final color of the first pixel is the directional lighting color. If only ambient lighting exists, the final color of the first pixel is the ambient lighting color. In the above calculation process, all material parameters involved in the calculation (albedo, metallicity, roughness) come from an optimizable material map, and the entire calculation process is differentiable, allowing the gradient of the rendering result relative to the material parameters to be accurately calculated and backpropagated.

[0077] After calculating the color of all first pixels, the second renderer organizes these pixels according to their position coordinates on the projection plane to form a complete 2D rendered image. The color value of each pixel in this rendered image is the final color information of the corresponding first pixel, while background pixels not covered by any triangular facets are set to default color values. Thus, the second renderer outputs a rendered image corresponding to the input camera pose, model pose, and lighting conditions. The color value of each pixel in the rendered image is derivative with respect to the second material parameters (i.e., pixel values ​​in the diffuse map, metallic map, and roughness map). This differentiable property allows for the calculation of the material parameter update gradient using backpropagation when there are differences between the rendered image and the target image, thereby achieving automatic optimization of the second material parameters.

[0078] In backpropagation, the gradient of the loss value is first propagated back to the PBR rendering equation to calculate the direction and magnitude of the material attribute adjustment needed for each pixel. Then, this gradient is propagated back to the material sampling stage, and based on the texture coordinates of each pixel, the error gradient is accurately accumulated in the gradient buffer of the corresponding pixel position in the material map. Finally, based on these accumulated gradients, the values ​​of multiple pixels in the material map are updated, thus completing one iteration of material parameter optimization. The texture coordinates can also be used to locate the corresponding sampling position in the second material parameter map during gradient backpropagation, and the material attributes are used to construct a differentiable computational link between color information and material parameters.

[0079] In summary, such as Figure 6 The differentiable rendering process shown inputs the second material parameters, camera pose, model pose, lighting conditions (en.hdr), and geometric data (mesh.obj) as differentiable rendering parameters into the second renderer. The second renderer parses the ambient light map using an HDR loader to generate lighting data, and then pre-filters the lighting data using a Mipmap generator. Simultaneously, it parses the geometric data file using an Obj loader to extract vertex coordinates, face indices, and normal information. All of this data is then input into the differentiable rendering pipeline. In the rasterization stage, the 3D geometric faces are projected onto a 2D plane, and the position attributes and texture coordinates of each pixel are obtained through differentiable rasterization. Then, based on the texture coordinates, the diffuse color, metallicity value, and roughness value of each pixel are sampled from the material map. Finally, based on these material properties, the final color of each pixel is calculated: for directional light, it is directly calculated using the PBR rendering equation; for ambient light, it is approximated using both diffuse and specular components. The ambient light and directional light calculation results are added together to obtain the final color of the pixel, forming the rendered image.

[0080] In some optional embodiments, the model can be split according to materials, and differentiable rendering and optimization can be performed on each individual material. Specifically, the geometric data of multiple material regions of the target 3D model are parsed from the target 3D model file, along with the first material parameters corresponding to each material region, and the second material parameters corresponding to each material region are initialized. For any material region in the target 3D model, rasterization, material sampling, and lighting calculations are performed independently based on its corresponding geometric patches and material parameters, and gradient backpropagation and parameter updates are completed within that material region. After the multiple material regions are optimized separately, the optimized material regions are then assembled back to the original model structure to restore the complete model. This approach significantly improves the accuracy of material restoration and avoids interference between different materials. Furthermore, it reduces the complexity of the differentiable rendering engine and fitting algorithm, minimizing dependence on and computational impact on geometric meshes, thereby improving overall optimization efficiency and stability.

[0081] Using the aforementioned material conversion method, each target 3D model is optimized to obtain target material parameters (diffuse reflection, metallicity, and roughness maps). These parameters, along with the geometric mesh data, are packaged into a target file usable by real-time rendering technology and stored. When needed, the target application can directly read the target file, automatically parse the target material parameters and geometric data using its built-in shaders, and instantiate and render the model in real-time within the application's rendering context. The model is presented in high-fidelity visual style on the application's canvas, allowing users to freely interact with it through mouse dragging, scrolling, and other operations. This achieves a complete closed loop from asset storage to real-time front-end rendering, meeting the needs of e-commerce displays, virtual showrooms, and other scenarios for instant loading and interactive presentation of 3D content.

[0082] Furthermore, this application embodiment also provides a three-dimensional model rendering method, including: acquiring a model file of a target three-dimensional model and multiple sets of scene parameters; parsing the geometric data of the target three-dimensional model and the first material parameters under a first renderer from the model file; performing forward rendering using the first renderer based on the geometric data, the first material parameters, and the multiple sets of scene parameters to obtain multiple reference images corresponding to the target three-dimensional model under the multiple sets of scene parameters; initializing the second material parameters used by a second renderer, and adapting the rendering path of the second renderer to the rendering path of the target application; performing differentiable rendering using the second renderer based on the geometric data, the second material parameters, and the multiple sets of scene parameters to obtain multiple rendered images corresponding to the target three-dimensional model under the multiple sets of scene parameters; updating the second material parameters based on the image reconstruction loss between the multiple rendered images and the multiple reference images, and re-performing differentiable rendering based on the updated second material parameters until the image reconstruction loss meets a preset convergence condition to obtain the target material parameters; and rendering the target three-dimensional model in the target application based on the target material parameters and the geometric data.

[0083] Furthermore, this application embodiment also provides a three-dimensional model rendering method, including: obtaining target material parameters, the target material parameters being obtained based on the material conversion method of the three-dimensional model described above; and rendering the target three-dimensional model in a target application in combination with geometric data based on the target material parameters, wherein the rendering path of the second renderer is adapted to the rendering path of the target application.

[0084] In summary, the material conversion method for a 3D model provided in this application first obtains a target 3D model, which includes geometric data and first material parameters, and acquires multiple sets of scene parameters. Based on this, a first renderer performs offline rendering on the geometric data and the first material parameters under multiple sets of scene parameters, obtaining multiple reference images. Since the first renderer is an embedded renderer in professional modeling software, its rendering results can accurately reflect the real visual effect of the original material under physical lighting conditions. Therefore, these reference images can serve as high-fidelity monitoring signals during the material conversion process. By introducing multiple sets of different scene parameters, the complete optical response characteristics of the first material parameters under diverse lighting conditions and viewing angles can be captured, providing rich and comprehensive training data for subsequent optimization and learning of material parameters. This effectively avoids the overfitting problem of material parameters caused by conversion in a single scene, and enhances the generalization ability and robustness of the converted material parameters under different environments.

[0085] Furthermore, a second renderer performs differentiable rendering on the geometric data and the material parameters to be converted under the same multiple sets of scene parameters, resulting in multiple rendered images. The rendering path of the second renderer is designed to adapt to the rendering path of the target application (such as a real-time rendering engine for a web application). That is, the second renderer simulates the target application's lighting model, shader calculation methods, and rendering pipeline, ensuring that the rendered images produced during the differentiable rendering process accurately reflect the actual display effects in the target application. This consistent rendering path design guarantees a WYSIWYG (What You See Is What You Get) process for material parameter optimization. In other words, when the optimized target material parameters are rendered in the target application, it reduces additional visual deviations caused by differences in rendering paths, effectively solving the technical problem of discrepancies between the converted effect and expectations due to different rendering environments.

[0086] Based on this, using multiple reference images as supervision signals, the total loss value between multiple rendered images and multiple reference images is calculated. The gradient backpropagation mechanism of differentiable rendering is then used to backpropagate the gradient of the loss value to the material parameters to be transformed, iteratively updating the material parameters until the loss function converges. Material fitting based on a differentiable rendering scheme aims to maximize the consistency of the rendered images from the perspective of the final material rendering result, possessing image-level quantitative evaluation capabilities. The optimization process also has rendering effect perception and material realism perception. Compared to traditional parameter or baking schemes, its final effect is more controllable and closer to the rendering result of offline rendering engines. Through end-to-end automatic optimization, there is no need for manual design of mapping rules or parameter adjustment strategies for different material types. It can automatically learn the mapping relationship from the first material parameter to the target material parameter, exhibiting strong generalization ability and scalability. It can adapt to different modeling software, different renderers, and 3D models with different material types, significantly reducing the cost of manual intervention and the technical threshold.

[0087] Figure 7 The figure illustrates the model rendering effects produced by the method provided in this application and conventional mainstream solutions. Several different types of models are listed in the figure, each rendered using a first renderer (e.g., Vray), a differentiable rendering scheme, and a mainstream scheme, respectively. Figure 7 As can be seen, the differentiable rendering solution is significantly better than the mainstream solution in terms of light and shadow feedback, detail representation and overall realism. It provides more realistic light and shadow feedback and its rendering effect is closer to the result of the first renderer.

[0088] Quantitatively, this application performs pixel-by-pixel comparison of two sets of rendered images based on the L1 algorithm: one set is a rendered image obtained in real-time in the target application based on optimized target material parameters, and the other set is a reference image obtained by rendering using a first renderer based on the first material parameters before optimization. The comparison images include rendered images from six perspectives: top, bottom, left, right, front, and back. First, the RGB data of the rendered images from each perspective are normalized, then the L1 loss of each of the three channels is calculated, and finally, the average value of all perspectives and channels is taken as the evaluation index. Experimental results show that after optimization by the embodiments of this application, the average L1 loss between the rendered image and the reference image decreased from 0.2401 to 0.1836, a relative improvement of 23.53%; the material qualification rate increased from 46.5% to 69.3%. Figure 8 As shown, the CDF (Cumulative Distribution Function) curve of the optimized score of the proposed solution is displayed. The horizontal axis represents the score, and the vertical axis represents the cumulative probability. The curve distribution shows that the score is mainly concentrated in the lower region, and the curve rises rapidly at the lower score and then flattens out. This indicates that the rendered image generated by the proposed embodiment has little difference from the reference image and exhibits high consistency, verifying the effectiveness of the proposed embodiment in material optimization.

[0089] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 15 can be device E; or the execution subject of steps 11 and 12 can be device E, and the execution subject of steps 13 to 15 can be device F, etc.

[0090] In some of the processes described in the above embodiments and accompanying drawings, multiple operations are included that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The sequence numbers of the operations are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0091] Figure 9 This is a schematic diagram of the structure of a material conversion device for a three-dimensional model provided in an embodiment of this application. Figure 9 As shown, the device includes: an acquisition module 91, a forward rendering module 92, a differentiable rendering module 93, and a gradient update module 94.

[0092] The module 91 is used to acquire the model file and multiple sets of scene parameters of the target 3D model, and parse the geometric data and first material parameters of the target 3D model under the first renderer from the model file. The forward rendering module 92 is used to perform forward rendering using the first renderer based on the geometric data, the first material parameters and the multiple sets of scene parameters to obtain multiple reference images of the target 3D model under the multiple sets of scene parameters. The differentiable rendering module 93 is used to initialize the second material parameters used by the second renderer, and the rendering path of the second renderer is adapted to the rendering path of the target application. Based on the geometric data, the second material parameters and the multiple sets of scene parameters, the second renderer performs differentiable rendering to obtain multiple rendered images of the target 3D model under the multiple sets of scene parameters. The gradient update module 94 is used to update the second material parameters based on the image reconstruction loss between the multiple rendered images and the multiple reference images, and to re-perform differentiable rendering based on the updated second material parameters until the image reconstruction loss meets the preset convergence condition to obtain the target material parameters. The target material parameters are used to render the target 3D model in the target application in combination with the geometric data.

[0093] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0094] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 3 As shown, the computing device includes a memory 101 and a processor 102.

[0095] Memory 101 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0096] Processor 102, coupled to memory 101, is used to execute computer programs in memory 101 for: acquiring model files and multiple sets of scene parameters of a target 3D model; parsing geometric data and first material parameters of the target 3D model from the model file using a first renderer; performing forward rendering using the first renderer based on the geometric data, first material parameters, and multiple sets of scene parameters to obtain multiple reference images of the target 3D model corresponding to the multiple sets of scene parameters; initializing second material parameters used by a second renderer, adapting the rendering path of the second renderer to the rendering path of the target application; performing differentiable rendering using the second renderer based on the geometric data, second material parameters, and multiple sets of scene parameters to obtain multiple rendered images of the target 3D model corresponding to the multiple sets of scene parameters; updating the second material parameters based on the image reconstruction loss between the multiple rendered images and the multiple reference images, and re-performing differentiable rendering based on the updated second material parameters until the image reconstruction loss meets a preset convergence condition to obtain target material parameters; wherein, the target material parameters are used to render the target 3D model in the target application in conjunction with geometric data.

[0097] Furthermore, such as Figure 10 As shown, the computing device also includes other components such as a communication component 103 and a power supply component 104.

[0098] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0099] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0100] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0101] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0102] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0103] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0104] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0105] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for material conversion of a three-dimensional model, characterized in that, include: Obtain the model file and multiple sets of scene parameters of the target 3D model, and parse the geometric data of the target 3D model and the first material parameters under the first renderer from the model file; Based on the geometric data, the first material parameters, and the multiple sets of scene parameters, the first renderer is used to perform forward rendering to obtain multiple reference images of the target 3D model under the multiple sets of scene parameters; Initialize the second material parameters used by the second renderer, and adapt the rendering path of the second renderer to the rendering path of the target application; Based on the geometric data, the second material parameters, and the multiple sets of scene parameters, the second renderer is used to perform differentiable rendering to obtain multiple rendered images of the target 3D model under the multiple sets of scene parameters. Based on the image reconstruction loss between the plurality of rendered images and the plurality of reference images, the second material parameters are updated, and differentiable rendering is performed again based on the updated second material parameters until the image reconstruction loss meets the preset convergence condition to obtain the target material parameters. The target material parameters are used to render the target 3D model in the target application in conjunction with the geometric data.

2. The method according to claim 1, characterized in that, Each set of scene parameters includes camera pose, model pose, and lighting conditions. Based on the geometric data, the second material parameters, and the multiple sets of scene parameters, the second renderer performs differentiable rendering to obtain multiple rendered images of the target 3D model corresponding to the multiple sets of scene parameters, including: For any set of scene parameters, input the geometric data, the second material parameters, and the scene parameters into the second renderer: The target 3D model is constructed based on the geometric data, and the target 3D model includes multiple geometric patches and their corresponding vertices; Based on the camera pose and model pose in any set of scene parameters, the plurality of geometric patches are projected and differentiable rasterized to obtain the first pixel point on the projection plane covered by the plurality of geometric patches. The material properties of the first pixel are sampled from the second material parameters based on the texture coordinates of the vertices corresponding to the geometric facets covering the first pixel. Calculate the color information of the first pixel based on the lighting conditions in any set of scene parameters and the material properties of the first pixel; The target 3D model is rendered using the first pixel and its texture coordinates, material properties, and color information under any set of scene parameters to obtain a rendered image.

3. The method according to claim 2, characterized in that, Based on the camera pose and model pose in any set of scene parameters, the plurality of geometric patches are projected and differentiable rasterized to obtain a first pixel point on the projection plane covered by the plurality of geometric patches, including: Based on the camera pose and model pose in any set of scene parameters, the vertices of the plurality of geometric patches are transformed from the model space to the camera space, and the transformed vertices are projected onto the projection plane to obtain the projection area corresponding to each of the plurality of geometric patches. The projection area is rasterized into a differentiable form to determine whether each pixel on the projection plane is covered by the projection area, and the pixel covered by the projection area is taken as the first pixel.

4. The method according to claim 2, characterized in that, Based on the texture coordinates of the vertices corresponding to the geometric facets covering the first pixel, the material properties of the first pixel are sampled from the second material parameters, including: Obtain the texture coordinates of the vertices corresponding to the geometric facet covering the first pixel and the centroid coordinates of the first pixel within the geometric facet; The texture coordinates of the first pixel are obtained by interpolating the texture coordinates of the vertex using the centroid coordinates; Based on the texture coordinates of the first pixel, the material properties of the first pixel are obtained by sampling from the second material parameters.

5. The method according to claim 4, characterized in that, The second material parameters include a diffuse map, a metallic map, and a roughness map. Based on the texture coordinates of the first pixel, the material properties of the first pixel are sampled from the second material parameters, including: Using the texture coordinates of the first pixel as sampling coordinates, diffuse color values ​​are sampled from the diffuse texture, metallic values ​​are sampled from the metallic texture, and roughness values ​​are sampled from the roughness texture. The diffuse color value, metallic value, and roughness value are combined to form the material properties of the first pixel.

6. The method according to claim 2, characterized in that, The lighting conditions include at least one directional light and / or ambient light. Based on the lighting conditions in any set of scene parameters and the material properties of the first pixel, the color information of the first pixel is calculated, including: For any directional light, calculate the directional light color generated by the directional light on the first pixel based on the material properties of the first pixel and the illumination direction and intensity of the directional light; For the ambient light, the ambient light color generated on the first pixel is calculated based on the material properties of the first pixel and the radiance contribution of the ambient light in multiple incident directions. The color information of the first pixel is obtained based on the directional light color generated by the at least one directional light on the first pixel and / or the ambient light color generated by the ambient light on the first pixel.

7. The method according to claim 6, characterized in that, The second material parameters include a diffuse map, a metallic map, and a roughness map. Based on the material properties of the first pixel and the radiance contribution of the ambient light in multiple incident directions, the ambient light color generated at the first pixel is calculated, including: The ambient light is processed to obtain an irradiance map, a pre-filtered environment map, and an integral lookup table. The integral lookup table is used to correct the contribution of the pre-filtered environment map to specular reflection based on roughness and viewing angle. The albedo is determined based on the diffuse reflection map, whereby the albedo represents the ability of an object's surface to diffusely reflect incident light. The base reflectivity is determined based on the metallicity map and the diffuse reflectivity map. The base reflectivity represents the proportion of energy that is directly specularly reflected when light is incident perpendicularly on the surface of an object. Calculate the diffuse color based on the irradiance map and the albedo; Calculate the specular reflection color based on the pre-filtered environment map, the integral lookup table, and the base reflectivity; The ambient light color is obtained by adding the diffuse color to the specular color.

8. The method according to any one of claims 1-7, characterized in that, Based on the geometric data, the first material parameters, and the multiple sets of scene parameters, forward rendering is performed using the first renderer to obtain multiple reference images of the target 3D model corresponding to the multiple sets of scene parameters, including: For any set of scene parameters, a rendering description file in the target format is generated based on the first material parameters, the geometric data, and the any set of scene parameters; The rendering description file is input into the first renderer, and the target 3D model is constructed based on the geometric data in the rendering description file. The target 3D model includes multiple geometric patches and their corresponding vertices. Based on the camera pose and model pose in the rendering description file, the plurality of geometric patches are projected onto the imaging plane and rasterized to obtain the second pixel point on the imaging plane covered by the plurality of geometric patches. Based on the first material parameters and lighting conditions in the rendering description file, the second pixel is shading calculated to obtain a reference image of the target 3D model under any set of scene parameters.

9. The method according to any one of claims 1-5, characterized in that, Also includes: Based on the parameter sampling rules, the preset parameter space is sampled to generate multiple sets of scene parameters. The preset parameter space includes a lighting condition parameter space, a camera pose parameter space, and a model pose parameter space. A set of scene parameters includes the corresponding camera pose, model pose, and lighting conditions.

10. A method for rendering a three-dimensional model, characterized in that, include: Obtain the model file and multiple sets of scene parameters of the target 3D model, and parse the geometric data of the target 3D model and the first material parameters under the first renderer from the model file; Based on the geometric data, the first material parameters, and the multiple sets of scene parameters, the first renderer is used to perform forward rendering to obtain multiple reference images of the target 3D model under the multiple sets of scene parameters; Initialize the second material parameters used by the second renderer, and adapt the rendering path of the second renderer to the rendering path of the target application; Based on the geometric data, the second material parameters, and the multiple sets of scene parameters, the second renderer is used to perform differentiable rendering to obtain multiple rendered images of the target 3D model under the multiple sets of scene parameters. Based on the image reconstruction loss between the plurality of rendered images and the plurality of reference images, the second material parameters are updated, and differentiable rendering is performed again based on the updated second material parameters until the image reconstruction loss meets the preset convergence condition to obtain the target material parameters. Based on the target material parameters, the target 3D model is rendered in the target application by combining the geometric data.

11. A method for rendering a three-dimensional model, characterized in that, include: Obtain target material parameters, wherein the target material parameters are obtained based on the method described in any one of claims 1-9; Based on the target material parameters, the target 3D model is rendered in the target application in combination with the geometric data, wherein the rendering path of the second renderer is adapted to the rendering path of the target application.

12. A computing device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable code that, when executed by a processor of a computing device, causes the processor to perform the method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 1 to 11.

Images