Method, device, equipment, medium and product for rendering three-dimensional model

Abstract

The application discloses a rendering method and device of a three-dimensional model, equipment, a medium and a product, and relates to the field of scene generation. The method comprises the following steps: acquiring a first three-dimensional model and rendering guide information corresponding to the first three-dimensional model; projecting the first three-dimensional model to obtain at least two projection geometries, wherein the at least two projection geometries correspond to different projection visual angles; fusing the rendering guide information and the at least two projection geometries respectively to generate model surface rendering information corresponding to the at least two projection geometries respectively; and rendering a model surface of the first three-dimensional model through the model surface rendering information corresponding to the at least two projection geometries respectively to obtain a second three-dimensional model. The model surface rendering information considering the correlation between multiple visual angles is generated by comprehensively fusing the rendering guide information and the at least two projection geometries, and the rendering precision of the second three-dimensional model and the coordination of the display effect are improved. The application can be applied to a game picture development scene, a film and animation production scene and the like.

Description

Technical Field

[0001] This application relates to the field of scene generation, and in particular to a method, apparatus, device, medium and product for rendering a three-dimensional model. Background Technology

[0002] With the development of computer technology, scene rendering has been used more and more widely. In order to improve the visual presentation effect, rendering technology has become more and more refined and advanced.

[0003] In related technologies, it is common to acquire the 3D model to be rendered and the text used to guide the rendering process. When analyzing multiple model faces of the 3D model separately, the text is used as a control signal, and a diffusion model is used to perform a denoising process on the progressively noisy model faces, thereby obtaining the texture information corresponding to each of the multiple model faces, so as to perform texture rendering on the 3D model to be rendered.

[0004] The poor correlation of texture information in the above techniques can easily lead to less than ideal model rendering accuracy after texture rendering, thus affecting the rendering effect. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for rendering a three-dimensional model. It can comprehensively generate model surface rendering information that considers the correlation between multiple projection viewpoints by integrating rendering guidance information and at least two projected geometry diagrams, thereby improving the rendering accuracy and display effect consistency of the second three-dimensional model. The technical solution is as follows.

[0006] On the one hand, a method for rendering a 3D model is provided, the method comprising:

[0007] Obtain the first 3D model and the rendering guidance information for the first 3D model;

[0008] The first three-dimensional model is projected to obtain at least two projected geometric figures, which correspond to different projection viewpoints and respectively represent different geometric shapes.

[0009] The rendering guidance information and the at least two projected geometry maps are respectively fused to generate model surface rendering information corresponding to the at least two projected geometry maps. The model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projection view.

[0010] The model surface of the first three-dimensional model is rendered using the model surface rendering information corresponding to the at least two projected geometry diagrams to obtain the second three-dimensional model.

[0011] On the other hand, a rendering apparatus for a three-dimensional model is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire the first 3D model and the rendering guidance information for the first 3D model;

[0013] The projection module is used to project the first three-dimensional model to obtain at least two projected geometric figures, wherein the at least two projected geometric figures correspond to different projection perspectives and respectively refer to different geometric shapes.

[0014] The fusion module is used to fuse the rendering guidance information and the at least two projected geometry maps respectively to generate model surface rendering information corresponding to the at least two projected geometry maps respectively. The model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projection view.

[0015] The rendering module is used to render the model surface of the first three-dimensional model using the model surface rendering information corresponding to the at least two projected geometry diagrams, to obtain the second three-dimensional model.

[0016] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the rendering method of a three-dimensional model as described in any of the embodiments of this application above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the rendering method of a three-dimensional model as described in any of the embodiments of this application above.

[0018] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the rendering method for the three-dimensional model described in any of the above embodiments.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] For the first 3D model, projected geometry maps obtained from at least two projection viewpoints are acquired. Rendering guidance information and these two projection geometry maps are then fused to generate model surface rendering information, which is used to render the surface of the first 3D model and obtain the second 3D model. During the process of guiding the display state of the first 3D model using rendering guidance information, corresponding projection geometry maps are obtained from at least two projection viewpoints. This process combines the rendering guidance information and the at least two projection geometry maps to generate model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two projection viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoints is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a structural block diagram of a rendering system provided in an exemplary embodiment of this application;

[0023] Figure 2 This is a flowchart of a rendering method for a three-dimensional model provided in an exemplary embodiment of this application;

[0024] Figure 3 This is a flowchart of a three-dimensional model rendering method provided in another exemplary embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the use of a multi-view diffusion model and an image transformation diffusion model to obtain model surface rendering information, provided in an exemplary embodiment of this application.

[0026] Figure 5 This is a flowchart of a three-dimensional model rendering method provided in another exemplary embodiment of this application;

[0027] Figure 6 This is an overall schematic diagram of a rendering method for a three-dimensional model provided in an exemplary embodiment of this application;

[0028] Figure 7 This is a schematic diagram of obtaining an RGB image from a model geometry from multiple perspectives, provided in an exemplary embodiment of this application.

[0029] Figure 8 This is a schematic diagram of obtaining a second three-dimensional model having a first display state, provided by an exemplary embodiment of this application;

[0030] Figure 9 This is a schematic diagram of an interface for changing the style of a 3D model provided in an exemplary embodiment of this application;

[0031] Figure 10 This is a structural block diagram of a rendering apparatus for a three-dimensional model provided in an exemplary embodiment of this application;

[0032] Figure 11 This is a structural block diagram of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] In related technologies, the 3D model to be rendered and the text used to guide the rendering process are typically acquired. When analyzing multiple facets of the 3D model separately, the text is used as a control signal, and a diffusion model is used to perform a denoising process on the progressively noisy facets, thereby obtaining texture information corresponding to each facet, so as to perform texture rendering on the 3D model to be rendered. In the above techniques, the correlation of texture information is poor, which can easily lead to less than ideal rendering accuracy of the model after texture rendering, affecting the rendering effect.

[0035] This application presents a method for rendering a 3D model, which integrates rendering guidance information and at least two projected geometry to generate model surface rendering information that considers the correlation between multiple projection viewpoints, thereby improving the rendering accuracy and display consistency of the second 3D model. The 3D model rendering method proposed in this application can be applied to various rendering scenarios that support 3D models, such as game development, film and animation production, architectural visualization and design, product design, industrial simulation, personalized design, virtual reality, and augmented reality, without limitation.

[0036] In an optional embodiment, the following description is given using the above-described 3D model rendering method as an example in a game screen development scenario.

[0037] In game graphics development, it's often necessary to design and render numerous 3D models as virtual elements within a virtual scene, such as 3D virtual props and buildings. After designing the first 3D model (e.g., a virtual axe), to enrich its display, rendering guidance information such as rendering text and style templates can be used to indicate the desired display state (e.g., display style) on the first 3D model. For example, the rendering guidance information could be the text "A colored axe."

[0038] This illustration demonstrates how a first 3D model, designed as a shape reference and given rendering guidance information as a style reference, can be used to perform texture baking on the first 3D model to enrich its rendering display style. For example, when using the first 3D model as a shape reference, projecting the first 3D model yields at least two projected geometric figures. These at least two projected geometric figures correspond to different projection viewpoints and represent different geometric shapes. For instance, the at least two projection viewpoints include at least two of the following: a frontal view, a rear view, a top view, and a side view. Projecting the first 3D model onto a preset projection plane along multiple projection viewpoints yields projected geometric figures corresponding to each of the at least two projection viewpoints. These projected geometric figures are the two-dimensional images projected under the respective projection viewpoints.

[0039] Next, the rendering guidance information and at least two projected geometry maps are fused to generate model surface rendering information corresponding to each of the at least two projected geometry maps. Each of the at least two projected geometry maps corresponds one-to-one with at least two projection viewpoints, and the model surface rendering information is used to render the display state of the model surface of the first 3D model under the projection viewpoints. Illustratively, under the constraints of the rendering guidance information, the projected geometry maps corresponding to at least two projection viewpoints are jointly analyzed to obtain model surface rendering information that conforms to both the rendering guidance information and the at least two viewpoints.

[0040] Finally, the model surface of the first 3D model is rendered using model surface rendering information corresponding to at least two projected geometry diagrams to obtain the second 3D model. Illustratively, model surface rendering information expresses the texture state of the model surface, such as albedo, roughness, and metallicity; at least two projection viewpoints correspond to the model surface rendering information. A baking process is performed on the model surface when viewing the first 3D model from the corresponding projection viewpoint using the model surface rendering information, thereby displaying the model surface rendering information on the first 3D model and obtaining the second 3D model. Compared to the first 3D model, the model surface of the second 3D model conforms to the model surface rendering information, and since the model surface rendering information is obtained by integrating multiple projection viewpoints, it contains viewpoint correlation. This allows the rendered second 3D model to more smoothly display the transformation relationship between multiple projection viewpoints and helps to maintain a relatively realistic display even under the influence of factors such as lighting changes, improving the display realism of virtual elements in the virtual scene.

[0041] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first three-dimensional model, rendering guidance information, and other content involved in this application were obtained with full authorization.

[0042] Secondly, the rendering system involved in the embodiments of this application will be described. The element rendering method provided in the embodiments of this application can be implemented by the terminal alone, by the server, or by the terminal and the server through data interaction. The embodiments of this application do not limit this. Optionally, the description will take the interaction between the terminal and the server to execute the element rendering method as an example.

[0043] This is illustrative; please refer to it. Figure 1 The rendering system involves a terminal 110 and a server 120, which are connected via a communication network 130.

[0044] In some embodiments, the terminal 110 has a data acquisition function for acquiring a first three-dimensional model and rendering guidance information for the first three-dimensional model.

[0045] Schematic illustration: A first 3D model is constructed using a model building application on terminal 110; or, terminal 110 downloads the first 3D model from the network; or, terminal 110 receives a first 3D model sent by another terminal; or, terminal 110 constructs the first 3D model using image data it has acquired itself, etc. Furthermore, terminal 110 can also obtain rendering guidance information for the first 3D model through information input operations of the model style adjustment object, such as: obtaining rendering text as rendering guidance information based on the text input operation of the object; or obtaining a style reference image as rendering guidance information based on the model import operation of the object, etc. Typically, the rendering guidance information indicates the display state of the adjusted first 3D model, such as the adjusted display style of the first 3D model, etc., which is not limited here.

[0046] In some embodiments, the terminal 110 performs the model rendering process based on the first 3D model and rendering guidance information; or, the terminal 110 sends the first 3D model and rendering guidance information to the server 120 through the communication network 130, and the server 120 performs the model rendering process, etc., without limitation. The following description takes the execution by the server 120 as an example.

[0047] In some embodiments, the server 120 projects the first three-dimensional model to obtain at least two projected geometries.

[0048] Among them, at least two projected geometric figures correspond to different projection perspectives and respectively refer to different geometric shapes.

[0049] Indicatively, at least two of a variety of projection perspectives, such as frontal view, rear view, top view, bottom view, left view, right view, etc., are used. In addition, the at least two perspectives may also include other perspectives (such as left rear view, etc.). The first three-dimensional model is projected from the selected at least two projection perspectives to obtain the projection geometry corresponding to the at least two projection perspectives respectively.

[0050] Projection is an analysis method that does not consider the depth of a 3D model. All model points of the 3D model are projected onto a preset projection plane along the projection direction. For example, taking at least two projection viewpoints, including the orthographic viewpoint, when projecting the first 3D model from the orthographic viewpoint, it means projecting the first 3D model along the depth axis (assuming the depth axis is the z-axis) onto a two-dimensional plane bounded by the horizontal and vertical coordinates (x-axis and y-axis), thus obtaining the orthographic viewpoint as the projection geometry corresponding to the orthographic viewpoint.

[0051] In some embodiments, the server 120 fuses the rendering guidance information and at least two projected geometry maps respectively to generate model surface rendering information corresponding to the at least two projected geometry maps respectively.

[0052] Among them, the model surface rendering information is used to render the display state of the model surface of the first 3D model under the projection view.

[0053] In a schematic way, using rendering guidance information as a display state reference and projected geometry obtained from at least two projection viewpoints as shape references, model surface rendering information containing multi-view correlation information and conforming to the rendering guidance information is generated. For example, if at least two projection viewpoints correspond to projection geometry, and each projection geometry corresponds to model surface rendering information, then model surface rendering information corresponding to at least two projection viewpoints is obtained.

[0054] In some embodiments, server 120 renders the model surface of the first three-dimensional model using model surface rendering information corresponding to at least two projected geometry diagrams, thereby obtaining the second three-dimensional model.

[0055] Indicatively, the second three-dimensional model is a three-dimensional model obtained by adjusting the surface of the first three-dimensional model from at least two projection perspectives.

[0056] Schematic representation: Model surface rendering information includes at least one of the following: albedo, roughness, metallicity, etc.; the model surface corresponding to each projected geometry represents the model surface observed from the projection viewpoint where the projected geometry is obtained. A baking process is performed on the model surface observed from the corresponding projection viewpoint using the model surface rendering information, thereby displaying the model surface rendering information on the first 3D model and obtaining the second 3D model. Because the generation process of the model surface rendering information fully considers the correlation between at least two viewpoints, it avoids inconsistency issues that may exist between different projection viewpoints, improves the rendering accuracy and display effect coordination of the second 3D model, and also helps to maintain a relatively realistic display even under the influence of factors such as changes in lighting, thus improving the realism of the second 3D model's display effect.

[0057] It is worth noting that the aforementioned terminals include, but are not limited to, mobile terminals such as mobile phones, tablets, portable laptops, smart voice interaction devices, smart home appliances, and in-vehicle terminals, and can also be desktop computers, etc.; the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers.

[0058] Based on the above introduction to terminology and application scenarios, the rendering method for the 3D model provided in this application will be explained, taking the application of this method to a server as an example. Figure 2 As shown, the method includes the following steps 210 to 240.

[0059] Step 210: Obtain the first 3D model and the rendering guidance information for the first 3D model.

[0060] In illustrative terms, a 3D model is a digital representation of the shape and structure of an object in three-dimensional space. Unlike a 2D image, a 3D model not only contains the shape of the object, but also information such as depth and spatial position, which allows the 3D model to be manipulated, rendered, and interacted with in three-dimensional space.

[0061] A 3D model typically includes model vertices (the basic building blocks of a 3D model, each with corresponding vertex coordinates in 3D space), model edges (connected by two model vertices, forming the boundary of the 3D model), model faces (areas enclosed by at least two model edges, usually forming a polygon to describe the surface of the 3D model), and normals (vectors used to describe the orientation of the model surface, helping to calculate rendering effects such as lighting and shadows). 3D models include at least one of several forms such as polygonal mesh models (e.g., triangular mesh models, quadrilateral mesh models), surface models, and voxel models.

[0062] Optionally, the first 3D model is a pre-defined 3D model that needs to be baked for rendering. Illustratively, certain rendering effects (such as lighting, shadows, reflections, etc.) can be pre-computed and saved to the texture map; this process is called "baking." The goal of baking rendering is to improve rendering efficiency, especially in scenes requiring real-time rendering (such as game scenes and virtual reality scenes), making the rendering process smoother and more efficient.

[0063] In illustrative terms, the first 3D model is a preset virtual 3D object (such as virtual props, virtual buildings, etc.) in the virtual world of the game scene. If it is necessary to bake and render the 3D object to make it display its effect more realistically in the virtual world, the preset virtual 3D object can be used as the first 3D model.

[0064] In some embodiments, the rendering guidance information is information used to adjust the display state of the first three-dimensional model. The display state includes the display style of the appearance of the first three-dimensional model, the display transparency of the appearance of the first three-dimensional model, etc. The display style includes at least one of a variety of styles such as color and brightness, which is not limited here.

[0065] In illustrative terms, rendering guidance information is used to guide the appearance of the first 3D model to be adjusted to the first display style. For example, if the first 3D model was originally in a blank style, the rendering guidance information is used to adjust the appearance of the first 3D model from the blank style to the first display style (such as red); or, if the first 3D model is currently in the previously selected second display style, the rendering guidance information is used to adjust the appearance of the first 3D model from the second display style to the first display style, and so on.

[0066] Optionally, the rendering guidance information is rendering text, which is a textual expression of the adjusted display state. For example, if the first 3D model is "a virtual axe" and the rendering text is "a colored axe", then the first adjusted display style represented by the rendering guidance information is "colored". This rendering guidance information is used to adjust the appearance of the first 3D model to "colored". Alternatively, the rendering guidance information is another 3D model with the first display style. This rendering guidance information is used to bake the first display style of the other 3D model onto the first 3D model so that its appearance changes to the first display style, etc. There is no limitation here.

[0067] Step 220: Project the first 3D model to obtain at least two projected geometric figures.

[0068] Indicatively, at least two projection viewpoints also represent different observation viewpoints. Each projection viewpoint typically corresponds to a virtual camera. The first three-dimensional model in three-dimensional space is observed from different virtual cameras, thus determining at least two projection viewpoints.

[0069] Among them, at least two projected geometric figures correspond to different projection perspectives and respectively refer to different geometric shapes.

[0070] Schematic, at least two projection viewpoints each correspond to a projection geometry. A projection geometry is formed by projecting the geometry of the first 3D model onto a 2D plane, showing the 2D projected shape observed from the corresponding projection viewpoint. That is, the first 3D model is a 3D solid model, and the projection geometry is the 2D plane image projected from the projection viewpoint. Therefore, the projection geometry is used to describe the shape, size, position, and features of the first 3D model in 3D space, as well as its characteristics after being projected onto the plane.

[0071] Projection is a shape analysis method that does not consider the depth of the three-dimensional model. During the projection process, all model points of the first three-dimensional model are projected onto a preset two-dimensional projection plane along the projection direction. That is, the projection position and projection direction of the two-dimensional projection plane are predefined. Projection directions include parallel projection, perspective projection, etc. The projection position is related to the projection angle, such as orthographic projection corresponding to the orthographic angle. Therefore, different projection angles correspond to different two-dimensional projection planes.

[0072] Optionally, the projection geometry corresponding to at least two projection viewpoints may be such as: the frontal projection geometry obtained by projecting from the frontal viewpoint along the parallel projection direction, the top view projection geometry obtained by projecting from the top viewpoint along the parallel projection direction, and the side view projection geometry obtained by projecting from the side viewpoint (such as the left side viewpoint, the right side viewpoint, etc.) along the parallel projection direction. The projection geometry may also be the projection geometry obtained by projecting from any other projection viewpoint, which is not limited here.

[0073] Optionally, different projection angles are used to project corresponding projection geometry diagrams. The geometric shapes represented by different projection geometry diagrams may be the same or different. For example, if the first 3D model is a cube, the projection geometry diagram corresponding to the frontal view is a square, and the projection geometry diagram corresponding to the top view is also a square, that is, the geometric shapes may be the same; or, if the first 3D model is a cone, the projection geometry diagram corresponding to the frontal view is a triangle, and the projection geometry diagram corresponding to the top view is a circle, that is, the geometric shapes may be different, which is not limited here.

[0074] Since the process of obtaining projected geometry from at least two projection perspectives for the first three-dimensional model can also be called a multi-view projection process, multi-view projection helps to clearly express the shape and position information of the first three-dimensional model and also helps to strengthen the correlation analysis between multiple projection perspectives.

[0075] Step 230: The rendering guidance information and at least two projected geometry maps are fused together to generate model surface rendering information corresponding to at least two projected geometry maps respectively.

[0076] Schematic, the rendering guidance information expresses the adjusted display state obtained by performing baking rendering on the first 3D model. Therefore, the rendering guidance information can be used as display state reference information. The projection geometry expresses the two-dimensional shape information of the first 3D model after projection from the corresponding projection viewpoint. Therefore, the projection geometry corresponding to at least two projection viewpoints can be used as shape reference information. By combining the rendering guidance information and at least two projection geometry, the effect influencing factors of the first 3D model in the rendering baking process can be predicted from at least two dimensions: display state and shape. These effect influencing factors can be called model surface rendering information.

[0077] Indicatively, at least two viewpoints correspond to projected geometry, and at least two projected geometry correspond to model surface rendering information, such as: viewpoint 1 corresponds to projected geometry A, and projected geometry A corresponds to model surface rendering information a, etc.

[0078] Among them, at least two projected geometry diagrams correspond one-to-one with at least two projected viewpoints; the model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projected viewpoints.

[0079] Schematic representation, at least two model face rendering information and at least two projected geometry Figure 1 One-to-one correspondence: at least two projected geometries correspond one-to-one with at least two projected viewpoints; therefore, at least two model surface rendering information correspond one-to-one with at least two projected viewpoints.

[0080] The display state refers to the appearance of the first three-dimensional model. The display state may include at least one of several states such as display style and display transparency. The model surface rendering information is the information obtained under the constraints of rendering guidance information and at least two projected geometry diagrams, which is used to adjust the model surface of the first three-dimensional model.

[0081] Optionally, the model surface rendering information includes at least one of albedo information, roughness information, and metallicity information.

[0082] Albedo information is usually represented by an albedo map. Albedo information characterizes the basic color of an object's (first 3D model) surface without considering factors such as lighting and shadows. It can also be understood as the material's inherent color. Albedo information includes at least one albedo value, and each albedo value usually corresponds to one pixel. Albedo values ​​are usually represented using three channels: red, green, and blue.

[0083] Roughness information is typically represented by a roughness map. Roughness information characterizes the smoothness of the microstructure of a 3D model's surface, controlling the degree of light reflection and / or scattering on the surface. Roughness information includes at least one roughness value, typically one pixel per roughness value. A roughness value is usually a number between 0 and 1, where 0 typically represents perfectly smooth (e.g., a mirror surface) and 1 typically represents very rough (e.g., a frosted surface).

[0084] Metallic information is typically represented by metallic maps. It characterizes whether a 3D model's material is metallic or non-metallic, and it interacts with albedo information. For example, the reflected color of a metal is usually close to its intrinsic material color as represented by its albedo information, while the reflected color of a non-metallic object is more influenced by the environment. Metallic information includes at least one albedo value, typically corresponding to one pixel. A metallic value is usually a number between 0 and 1, where 0 generally represents non-metallic and 1 generally represents fully metallic.

[0085] Optionally, each projection viewpoint corresponds to a projection geometry map and a model surface rendering information map. A model surface rendering information map includes at least one of the following: an albedo map, a roughness map, and a metallicity map of the model surface (usually a portion of the overall model surface of the first 3D model) when viewed from the projection viewpoint. For example, projection viewpoint 1 corresponds to projection geometry map A, and the model surface rendering information 'a' corresponding to projection geometry map A includes: an albedo map 'a1', a roughness map 'r1', and a metallicity map 'm1' of the model surface corresponding to the projection geometry map.

[0086] To illustrate, taking the use of rendering guidance information to adjust the display state of the first 3D model as an example, the first display style, as the overall style to be displayed on the surface of the first 3D model, may exhibit different effects on the projected geometry under different projection viewpoints corresponding to the first 3D model. There is a correspondence between model surface rendering information and projected geometry. The projected geometry corresponding to each projection viewpoint typically indicates a portion of the model surface on the first 3D model. The model surface rendering information is used to perform a baking rendering process on the portion of the model surface corresponding to the projected geometry. For example, for the portion of the model surface corresponding to projected geometry A, the albedo map a1, roughness map r1, and metallicity map m1 included in the model surface rendering information a are used as influencing factors in the baking rendering process to perform a baking rendering process on that portion of the model surface.

[0087] Step 240: Render the model surface of the first three-dimensional model using the model surface rendering information corresponding to at least two projected geometry diagrams to obtain the second three-dimensional model.

[0088] Indicatively, the model surface rendering information includes at least one of albedo information, roughness information, and metallicity information. For example, the model surface rendering information corresponding to each projection viewpoint is at least one of albedo map, roughness map, and metallicity map. The model surface rendering information corresponding to each projection viewpoint provides color, detail, gloss, etc. to the model surface by mapping the two-dimensional model surface rendering information (also known as two-dimensional texture image, two-dimensional texture, etc.) onto the model surface of the first three-dimensional model.

[0089] In a schematic way, the entire surface of the first 3D model is unfolded into a 2D model plane. This process can be called UV unwrapping (where U represents the horizontal coordinate and V represents the vertical coordinate) or UV mapping. The purpose of UV mapping is to assign a 2D coordinate (UV coordinate) to each model vertex. Then, the 2D model surface rendering information (such as albedo map, metallic map, roughness map, etc.) is mapped onto the part of the model surface in the first 3D model that corresponds to the model surface rendering information, thereby adding surface details to the first 3D model.

[0090] Indicatively, the second three-dimensional model is a three-dimensional model obtained by adjusting the surface of the first three-dimensional model from at least two projection perspectives.

[0091] In a schematic way, the model surface corresponding to the projection viewpoint in the first 3D model is baked and rendered using model surface rendering information to obtain the second 3D model after the display state is adjusted, such as adjusting the blank second 3D model to a red second 3D model. Therefore, compared with the first 3D model, the shape of the second 3D model usually remains unchanged. The second 3D model is the model obtained by adjusting the first 3D model in other display states (such as blank style, other styles, etc.) to the first display state, which is the style indicated by the rendering guidance information.

[0092] Optionally, since the model rendering information is at least one of information such as albedo information, roughness information, and metallicity information, the model surface rendering information better reflects the material state of the model surface that conforms to the model rendering information. Therefore, the second three-dimensional model obtained by rendering the first three-dimensional model through the model surface rendering information not only contains the appearance of the model, but also contains material properties that can change with the lighting. That is, the model surface rendering information can affect the display effect of the model surface of the second three-dimensional model according to the change of lighting conditions, making the rendering result more realistic and dynamic.

[0093] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0094] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0095] In an optional embodiment, since the model surface rendering information is obtained by fusing rendering guidance information and projected geometry obtained from at least two projection viewpoints, where each projected geometry corresponds to a separate model surface rendering information, the acquisition of each model surface rendering information is achieved by integrating the projected geometry corresponding to at least two projection viewpoints under the constraints of the rendering guidance information. Therefore, the acquisition process of model surface rendering information considers the correlation between multiple viewpoints, making the model surface rendering information more balanced. (Illustrative example, such as...) Figure 3 As shown above, Figure 2 Step 230 shown can also be implemented as steps 310 to 320.

[0096] Step 310: Fuse the rendering guidance information and the i-th projection geometry in at least two projection geometry maps to generate the model surface color image corresponding to the i-th projection viewpoint.

[0097] Where i is a positive integer, the model surface color image corresponding to the i-th projection viewpoint is used to describe the surface color of the model surface observed from the i-th projection viewpoint under the first lighting condition, and the rendering guidance information corresponds to the first lighting condition.

[0098] Schematic: If the model surface color image is executed from at least two different projection viewpoints, the rendering guidance information and at least two projection geometry maps can be fused to generate model surface color images corresponding to at least two projection viewpoints respectively. In this case, the i-th projection viewpoint is any one of the at least two projection viewpoints, and the model surface color image corresponding to the i-th projection viewpoint can be called the i-th model surface color image; the i-th projection viewpoint corresponds to the i-th projection geometry map, and the model surface under the i-th projection viewpoint is the model surface corresponding to the i-th projection geometry map.

[0099] Indicatively, each projection viewpoint corresponds to a projection geometry. Each projection geometry can be fused to obtain the model surface color image under the corresponding projection viewpoint. The model surface color image is the color displayed under the corresponding projection viewpoint.

[0100] As the rendering guidance information represents the first display state, although the model surface color image can present the colors of the first display state under the known lighting conditions (such as omnidirectional uniform lighting conditions) expressed by the rendering guidance information, considering that the rendering display process can also be affected by environmental factors such as lighting, reflection, scattering, and roughness, directly rendering the model surface color image corresponding to the projection viewpoint onto the surface of the first 3D model will easily result in the rendered 3D model only showing a static color display, with poor richness in terms of changes with lighting and other environmental factors, i.e., the display method is relatively limited. Therefore, the model surface color image is an intermediate analysis result under the constraints of the first display state expressed by the rendering guidance information, and further detailed analysis of the model surface color image is required afterwards.

[0101] In an optional embodiment, the information feature representation corresponding to the rendering guidance information is obtained.

[0102] In a schematic manner, the rendering guidance information is processed through a feature extraction network to obtain a feature representation corresponding to the rendering guidance information. The feature extraction network is at least one of various neural networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Encoders, and Transformers.

[0103] In an optional embodiment, the geometric feature representation corresponding to the i-th projected geometry is obtained.

[0104] To illustrate, for any i-th projected geometry in at least two projected geometry maps, a feature extraction process is performed on the i-th projected geometry map through a feature extraction network to obtain the geometric feature representation corresponding to the i-th projected geometry map.

[0105] Similarly, the projected geometry corresponding to at least two projection viewpoints is processed through a feature extraction network to obtain geometric feature representations corresponding to the at least two projected geometry, which can also be called geometric feature representations corresponding to the at least two projection viewpoints. The feature extraction network is at least one of various neural networks such as convolutional neural networks, recurrent neural networks, encoders, and converters.

[0106] In an optional embodiment, the information feature representation and the geometric feature representation corresponding to the i-th projected geometry are fused to obtain the i-th first feature representation.

[0107] Optionally, for the geometric feature representation corresponding to the i-th projected geometry, feature fusion (such as feature concatenation) can be performed on the information feature representation and the geometric feature representation corresponding to the i-th projected geometry to obtain the i-th first feature representation corresponding to the i-th projected geometry.

[0108] Optionally, if geometric feature representations corresponding to at least two projected geometry maps are obtained, feature fusion can be performed on the information feature representation corresponding to the rendering guidance information and the geometric feature representations corresponding to the at least two projected geometry maps during the process of fusing the rendering guidance information and the projected geometry maps obtained from the projection of at least two projection viewpoints, to obtain a first fused feature representation. The first fused feature representation includes feature information of the first feature representations corresponding to the at least two projected geometry maps.

[0109] In some embodiments, geometric feature representations corresponding to at least two projected geometric figures are obtained; feature concatenation is performed on the at least two geometric feature representations to obtain a geometric concatenation feature representation.

[0110] In a schematic manner, a feature extraction network is used to extract geometric feature representations corresponding to at least two projected geometric images; these representations are then concatenated to obtain a geometrically concatenated feature representation. Each of the at least two geometric feature representations represents the shape information displayed by the projected geometric image from a single projection viewpoint. Therefore, the concatenated geometrically concatenated feature representation can comprehensively represent shape information from multiple viewpoints, improving the correlation between multiple projection viewpoints.

[0111] In some embodiments, cross-attention analysis is performed on the information feature representation and the geometric splicing feature representation to obtain a first fused feature representation.

[0112] To illustrate, attention analysis is a method for analyzing the feature relationships between feature representations, while cross-attention analysis is a method for analyzing the feature relationships between at least two different feature representations. Optionally, during cross-attention analysis, the mutual influence between at least two feature representations is calculated according to some rule (such as dot product similarity) to quantify the relationship between feature representations.

[0113] In a schematic manner, cross-attention analysis is performed on the geometric splicing feature representation containing multi-view shape information and the information feature representation corresponding to the rendering guidance information, in order to achieve the purpose of feature correlation analysis between the information feature representation and the geometric splicing feature representation.

[0114] As shown in Formula 1 below, this is an illustrative calculation formula used in attention processing.

[0115] Formula 1:

[0116]

[0117] Where Q represents the query vector, used to indicate the information that is desired to be "queried" or "watched"; K is the key vector; K T The transpose matrix representing the key vectors; QK T is the dot product of the transposes of the query vector and the key vector, representing the similarity between the query vector and the key vector; V is the value vector, ultimately used for weighted output; in the attention mechanism, the query vector usually comes from the decoder, while the key vector and value vector usually come from the encoder (i.e., in the attention mechanism, the query vector, key vector, and value vector usually come from different sources); d k is the dimension of the key vector, used to scale the dot product result to prevent the gradient from vanishing due to an excessively large dot product; softmax is a normalization operation used to transform the dot product between Q and K into a probability distribution, and its output is a weight representing the degree of attention. These weights are used to weight the vector to obtain the final output.

[0118] Optionally, if the key vector K and value vector V are the same under the cross-attention mechanism, then Attention(Q,K,V) can be rewritten as Attention(Q,V,V), which is the calculation formula for the cross-attention mechanism.

[0119] In a schematic manner, during the cross-attention analysis of the information feature representation and the geometric concatenation feature representation, a linear transformation is performed on the information feature representation to obtain a first query vector, a first key vector, and a first value vector; and a linear transformation is performed on the geometric concatenation feature representation to obtain a second query vector, a second key vector, and a second value vector. Substituting the first query vector, the second key vector, and the second value vector into Formula 1 above, a first cross-attention feature representation with the information feature representation as a reference and the geometric concatenation feature representation as a comparison is obtained. Substituting the second query vector, the first key vector, and the first value vector into Formula 1 above, a second cross-attention feature representation with the geometric concatenation feature representation as a reference and the information feature representation as a comparison is obtained. Concatenating the first cross-attention feature representation and the second cross-attention feature representation yields a first fused feature representation.

[0120] In some embodiments, cross-attention analysis is performed on the information feature representation and the geometric splicing feature representation to obtain an attention splicing feature representation, which includes attention sub-feature representations corresponding to at least two geometric feature representations respectively; self-attention analysis is performed on the attention sub-feature representations corresponding to at least two geometric feature representations respectively to obtain self-attention feature representations corresponding to at least two attention sub-feature representations respectively.

[0121] Among them, the self-attention feature representations corresponding to at least two attention sub-feature representations include the self-attention feature representation corresponding to the i-th projection viewpoint, and the self-attention feature representation corresponding to the i-th projection viewpoint serves as the i-th first feature representation corresponding to the i-th projection viewpoint.

[0122] Indicatively, geometric splicing feature representation is a feature representation obtained by integrating geometric information from multiple projection perspectives. Cross-attention analysis is performed on the information feature representation and the geometric splicing feature representation, enabling the attention splicing feature representation to exhibit stronger feature correlation.

[0123] Since the geometric splicing feature representation is obtained by splicing at least two geometric feature representations, the corresponding cross-analysis will also be performed based on the position of the feature representation during the cross-attention analysis process. Therefore, the attention splicing feature representation also includes attention sub-feature representations corresponding to at least two geometric feature representations respectively.

[0124] The role of self-attention is to dynamically adjust the weights of feature representations at different positions when processing sequence data, thereby better capturing global contextual information.

[0125] Indicatively, the attention splicing feature representation includes at least two attention sub-feature representations, each of which corresponds one-to-one with the geometric feature representations corresponding to at least two projected geometries; after performing self-attention analysis on the at least two attention sub-feature representations, at least two self-attention feature representations are obtained, each of which corresponds one-to-one with the at least two self-attention feature representations.

[0126] For any i-th projection view among at least two projection views, the i-th projection view corresponds to the i-th geometric feature representation; based on cross-attention analysis, the i-th projection view also corresponds to the i-th attention sub-feature representation; based on self-attention analysis, the i-th projection view also corresponds to the i-th self-attention feature representation.

[0127] Therefore, in the process of fusing information feature representation and geometric feature representation corresponding to the i-th projected geometry, not only is the correlation between multiple feature representations integrated based on cross-attention analysis, but also the feature representation itself is analyzed in depth from a global perspective based on self-attention analysis, so that the i-th self-attention feature representation can better represent the predicted observation situation under the i-th projection perspective from a global dimension.

[0128] The i-th self-attention feature representation under the i-th projection view is taken as the i-th first feature representation under the i-th projection view; similarly, the self-attention feature representations corresponding to different projection views can be taken as the first feature representations of the corresponding projection views.

[0129] Optionally, the first geometric feature in the geometric splicing feature representation is represented at the first splicing position; the first attention sub-feature in the attention splicing feature, which corresponds to the first geometric feature representation, is spliced ​​at the second splicing position corresponding to the first splicing position.

[0130] Schematic illustration: an attention feature representation includes at least two attention sub-feature representations, and a geometric splicing feature representation includes at least two geometric feature representations; there is a one-to-one correspondence between the at least two attention sub-feature representations and the at least two geometric feature representations, and the feature position of the attention sub-feature representation in the attention feature representation is determined based on the feature position of the corresponding geometric feature representation in the geometric splicing feature representation. That is: based on the feature position of the geometric feature representation in the geometric splicing feature representation, the attention sub-feature representation corresponding to the geometric feature representation is represented at the corresponding feature position in the attention feature representation.

[0131] Self-attention analysis enables each attentional sub-feature in the attention feature representation to comprehensively consider the influence of other attentional sub-feature representations during the process of obtaining its corresponding self-attention feature representation. This allows for the capture of complex dependencies between each attentional sub-feature representation from a global perspective, resulting in a self-attention feature representation with global information. For example, if the attention feature representation includes attentional sub-feature representation 1 and attentional sub-feature representation 2, self-attention analysis allows attentional sub-feature representation 1 to not only focus on itself but also on attentional sub-feature representation 2 (e.g., dynamically adjusting its weight based on the importance of each attentional sub-feature representation), thus outputting the self-attention feature representation 'a' corresponding to attentional sub-feature representation 1. Similarly, self-attention analysis allows attentional sub-feature representation 2 to not only focus on itself but also on attentional sub-feature representation 1, thus outputting the self-attention feature representation 'b' corresponding to attentional sub-feature representation 2.

[0132] In some embodiments, a first fused feature representation is obtained by fusing the self-attention feature representations corresponding to at least two attention sub-feature representations respectively.

[0133] Optionally, the process of obtaining the first fused feature representation may include not only the cross-attention analysis process described above, but also the self-attention analysis process described above. Illustratively, the first fused feature representation is obtained by concatenating the self-attention feature representations corresponding to at least two attention sub-feature representations.

[0134] In addition to performing correlation analysis between the geometric splicing feature representation after fusing the above at least two geometric feature representations and the information feature representation to obtain the first fused feature representation, the information feature representation and at least two geometric feature representations can also be correlated separately to obtain the first fused feature representation. The following illustrates another illustrative method for obtaining the first fused feature representation, which also represents the relationship between geometric feature representations and information feature representations under different projection perspectives.

[0135] In some embodiments, cross-attention analysis is performed on the geometric feature representation and information feature representation corresponding to the i-th projected geometry to obtain a first attention feature representation corresponding to the i-th geometric feature representation.

[0136] Schematic illustration: Since rendering guidance information can be viewed as guidance constraint information for analyzing the projection geometry corresponding to at least two projection viewpoints, the relationship between at least two geometric feature representations and the information feature representation can be analyzed with reference to strengthen the correlation between the geometric feature representations and the information feature representations. That is, strengthening the correlation between the projection geometry and rendering guidance information (such as representing the first display state). For example, the geometric feature representation corresponding to the i-th projection geometry is any one of the at least two geometric feature representations; analyzing the relationship between the geometric feature representation corresponding to the i-th projection geometry and the information feature representation yields the first attention feature representation corresponding to the i-th projection geometry.

[0137] Optionally, cross-attention analysis can be used to achieve the purpose of correlation analysis between feature representations. Illustratively, for any geometric feature representation and the information feature representation corresponding to the rendering guidance information, the feature correlation between the geometric feature representation and the information feature representation is analyzed according to the attention formula shown in Formula 1 above, thereby obtaining the first attention feature representation corresponding to the geometric feature representation. The above analysis process is repeated to obtain the first attention feature representations corresponding to at least two geometric feature representations respectively; this will not be elaborated further here.

[0138] Here, the i-th first attention feature representation is used to characterize the feature association relationship between the i-th geometric feature representation and the information feature representation. Therefore, there are at least two first attention feature representations, and at least two first attention feature representations correspond one-to-one with at least two geometric feature representations. Each first attention feature representation characterizes the feature association relationship between its corresponding geometric feature representation and information feature representation.

[0139] In some embodiments, geometric feature representations corresponding to at least two projected geometry maps are obtained, including the geometric feature representation corresponding to the i-th projected geometry map; self-attention analysis is performed on the first attention feature representations corresponding to the at least two geometric feature representations to obtain second attention feature representations corresponding to the at least two first attention feature representations.

[0140] In illustrative terms, referring to the above method of performing self-attention analysis on attention sub-feature representations, self-attention analysis is performed on at least two first attention feature representations to analyze the correlation between the at least two first attention feature representations, thereby obtaining second attention feature representations corresponding to the at least two first attention feature representations respectively.

[0141] Schematic, at least two geometric feature representations each correspond to one first attention feature representation, thus there exist at least two first attention feature representations, and at least two first attention feature representations correspond one-to-one with at least two geometric feature representations. Self-attention analysis is performed on at least two first attention feature representations such that each first attention feature representation corresponds to one second attention feature representation, thus obtaining second attention feature representations corresponding to at least two first attention feature representations respectively.

[0142] The second attention feature representation is an enhanced feature representation based on at least two first attention feature representations. Illustratively, the purpose of self-attention analysis is to obtain enhanced features by combining the global information expressed by at least two first attention feature representations with the corresponding first attention feature representations.

[0143] Therefore, cross-attention analysis is first performed on the information feature representations corresponding to the geometric feature representations and the rendering guidance information to better capture the dependency relationship between them, enabling the first attention feature representation to characterize their correlation. Furthermore, compared to the first attention feature representation, the second attention feature representation has a more comprehensive information expression capability, improving the accuracy of subsequent acquisition of model surface color images. Specifically, the second attention feature representation corresponding to the i-th projection viewpoint serves as the i-th first feature representation corresponding to the i-th projection viewpoint.

[0144] In some embodiments, at least two first attention feature representations are fused with their corresponding second attention feature representations to obtain a first fused feature representation.

[0145] Schematic, at least two second attention feature representations are concatenated to obtain a first fused feature representation.

[0146] In an optional embodiment, the model surface color image corresponding to the i-th projection viewpoint is generated based on the i-th first feature representation.

[0147] Optionally, at least two model surface color images corresponding to the first fusion feature representation are generated.

[0148] Schematic illustration: Since the acquisition of the first fused feature representation involves comprehensive analysis of both the information feature representation and the geometric feature representations corresponding to at least two projection views, the first fused feature representation encompasses the information corresponding to at least two projection views under the constraints of the information feature representation. For example, the first fused feature representation includes first feature representations corresponding to at least two projection views, each of which represents a projection view. For instance, the first feature representation corresponding to the i-th projection view might be either the second attention feature representation or the self-attention feature representation corresponding to the i-th projection view, as described above; no specific limitation is imposed here.

[0149] Optionally, for the first feature representations in the first fused feature representations that correspond to at least two projection viewpoints respectively, the model surface color images corresponding to at least two first feature representations are obtained through feature decoding processing, that is, the model surface color images corresponding to at least two projection viewpoints are generated respectively.

[0150] Optionally, feature decoding processing is performed on the first fused feature representation to obtain the model surface color model corresponding to the first fused feature representation. The model surface color model includes model surface color images corresponding to at least two projection viewpoints, such as the model surface color model being a restored three-dimensional color model, or the model surface color image being a two-dimensional image of the three-dimensional color model under at least two projection viewpoints, etc., which are not limited here.

[0151] By analyzing each projection viewpoint separately and fusing the information feature representation with the geometric feature representation of the projection geometry corresponding to that viewpoint, the fused first feature representation can reveal the deep shape information represented by the projection geometry under the constraints of rendering guidance information. Furthermore, the model surface color image is obtained through the high-level first feature representation, achieving the goal of obtaining a high-quality model surface color image from the constraints of semantic and shape dimensions, thus ensuring image quality.

[0152] In some embodiments, a multi-view diffusion model and a first noise image are obtained.

[0153] Among them, the multi-view diffusion model is a pre-trained model used to obtain the color image corresponding to the projection viewpoint; the multi-view diffusion model can also be called the multi-view stable diffusion model.

[0154] Indicatively, a multi-view diffusion model is a stable diffusion model trained through comprehensive analysis of images from multiple viewpoints. Stable diffusion models are a type of generative model used to generate high-quality images, audio, and other data. Their basic idea is to restore the original data by simulating the gradual destruction (adding noise) of the data and then performing a reverse restoration process. That is, the diffusion model learns iteratively how to gradually "de-noise" the data to generate it. Therefore, the pre-trained multi-view diffusion model also involves a denoising process during application.

[0155] Optionally, the first noisy image is typically implemented as a random noisy image in the application of the multi-view diffusion model; alternatively, the first noisy image can also be a noisy image obtained by adding noise to the geometric figure; or, if the rendering guidance information is implemented as a guidance image (such as a two-dimensional image of a three-dimensional guidance model under a certain projection viewpoint), then the first noisy image can also be a noisy image obtained by adding noise to the guidance image, etc., without limitation here. The purpose of obtaining the first noisy image is to perform denoising processing on the first noisy image through the multi-view diffusion model.

[0156] In some embodiments, the first feature is represented as the first denoising condition corresponding to the first projection viewpoint, and the first noise image is denoised by a multi-view diffusion model to obtain the model surface color image corresponding to the first projection viewpoint.

[0157] Optionally, the first noisy image is a two-dimensional image. During the process of using the i-th first feature representation as the first denoising condition for the i-th projection viewpoint, the first feature representation is obtained under the constraints of rendering guidance information and multiple projection viewpoints. Therefore, using the first noisy image as the denoising condition has stronger accuracy. Thus, higher quality model surface color images corresponding to at least two projection viewpoints can be obtained through denoising processing. That is, the i-th first feature representation is used as the first denoising condition to obtain the model surface color image under the i-th projection viewpoint, where i is a positive integer.

[0158] To illustrate, if we want to obtain model surface color images corresponding to at least two projection viewpoints, taking the model surface color image corresponding to the i-th projection viewpoint as an example, we use the i-th first feature representation as the first denoising condition to perform a denoising process, thereby obtaining the model surface color image corresponding to the i-th first feature representation under the i-th projection viewpoint. The denoising process obtains the model surface color image by progressively generating detail information, which also improves the image quality of the model surface color image. Furthermore, the different model surface color images obtained by using the first feature representations corresponding to different projection viewpoints as denoising conditions also exhibit strong consistency and correlation.

[0159] In some embodiments, a first fusion feature is used as a first denoising condition, and a first noisy image is denoised using a multi-view diffusion model to obtain a first multi-view image.

[0160] Schematic, the first denoising condition is the denoising condition used by the multi-view diffusion model when performing denoising processing on the first noisy image. The guidance of the first denoising condition can make the denoising process biased. Since the first fusion feature representation contains rendering guidance information and the shape information of the model geometry corresponding to at least two views, the first fusion feature representation is used as the first denoising condition, so that the denoising process tends to obtain an image that conforms to the rendering guidance information and has the corresponding model geometry representation shape in at least two projection views, that is, the first multi-view image obtained through denoising processing.

[0161] The first multi-view image includes model surface color images corresponding to at least two projection viewpoints.

[0162] Schematic, for the first noisy image (if the first noisy image is a single noisy image, it can be reused; or, first noisy images corresponding to at least two projection viewpoints can be obtained, etc., each first noisy image can be the same random noise image, etc.), each first feature representation in the first fusion feature representation is used as the first denoising condition, and denoising processing is performed on the first noisy image to obtain model surface images corresponding to at least two projection viewpoints. For example, based on the number of viewpoints of the selected at least two projection viewpoints, a corresponding number of random noise images (which can be the same or different) are obtained as the first noisy image. If the number of viewpoints is 4, then 4 random noise images are obtained as the first noisy image (or if the number of viewpoints is 6, then 6 random noise images are obtained, which is not limited here). For the first noisy image N1, the first feature representation corresponding to projection viewpoint 1 in the first fusion feature representation is used as the first denoising condition, and denoising processing is performed on the first noisy image N1 to obtain the model surface color image RGB1 corresponding to projection viewpoint 1. Similarly, model surface color images corresponding to at least two projection viewpoints can be obtained. The combined at least two model surface color images can be called the first multi-view image.

[0163] Normally, the denoising process is achieved through the above-mentioned two-dimensional first noisy image. However, the first noisy image can also be realized as a three-dimensional image. In the process of using the first fusion feature representation as the first denoising condition, the three-dimensional first multi-view image constrained by the first fusion feature representation is obtained through denoising. The first multi-view image includes model surface color images corresponding to at least two projection viewpoints, which are not limited here.

[0164] Step 320: Based on at least two model surface color images, generate model surface rendering information corresponding to at least two projected geometry images.

[0165] Indicatively, at least two model surface color images correspond one-to-one with at least two projection viewpoints. The model surface color images represent the predicted information observed by at least two projection viewpoints under fixed environmental conditions (such as the first lighting condition) under rendering guidance information.

[0166] Therefore, the model surface color image provides relatively objective information while ignoring the effects of lighting, eliminating the interference of lighting on the display state of the 3D model. Based on the model surface color image, in order to fully consider the possible changes in display effects when environmental conditions change (e.g., although the overall first display state represented by the rendering guide information remains unchanged, subtle changes in reflection, scattering, etc., may occur under different lighting conditions), it is necessary to obtain deeper texture attributes from the model surface color image as model surface rendering information, so as to perform the subsequent baking rendering process based on the model surface rendering information.

[0167] Optionally, for the model surface color images corresponding to at least two projection viewpoints, model surface rendering information corresponding to each projection viewpoint is generated.

[0168] In illustrative terms, model surface rendering information includes at least one of albedo information, roughness information, and metallicity information; albedo information is estimated by analyzing the model surface color image through the lighting model, roughness information is estimated by the distribution of details, highlights, and reflections on the model surface in the model surface color image, and metallicity information is estimated by the difference between reflected color and highlight brightness, etc., without limitation here.

[0169] In some embodiments, a pre-trained lighting model is obtained, such as at least one of the following: the Phong Reflection Model, the Lambert Reflection Model, etc.; wherein, under the Phong Reflection Model, the brightness of the reflected light is related to the angle between the light source and the normal to the model surface (i.e., diffuse reflection), so it is generally applicable to ideal matte surfaces; under the Lambert Reflection Model, specular reflection (highlight reflection) is considered in addition to diffuse reflection.

[0170] For illustration purposes, it is assumed that the model surface is mainly diffuse (e.g., the reflection is assessed according to the type of the first 3D model, such as a virtual house, virtual axe, etc., then the model surface is considered to be mainly diffuse). Since albedo characterizes the relationship between surface color and light source intensity, assuming that the light source is uniform and the surface material is uniform, the albedo information can be calculated using the RGB value of each pixel in the model surface color image and the intensity of the light source, as shown in Formula 2 below.

[0171] Formula 2:

[0172]

[0173] Here, Albedo represents the albedo value corresponding to each pixel in the albedo information; ObservedReflectance is the color of the visible pixels in the model surface color image (i.e., the brightness of the reflection); and LightIntensity is the known light source intensity (such as the light source intensity known in the rendering guide information).

[0174] By performing the analysis shown in Formula 2 on multiple pixels, the albedo value of each pixel can be determined, thereby obtaining the albedo information corresponding to the color image of the model surface.

[0175] In some embodiments, metallicity information describes whether the model surface is metallic. Metallic and non-metallic surfaces have different reflective properties. Metallic surfaces exhibit strong specular reflection, and the reflected color is consistent with the original color of the metallic surface. Non-metallic surfaces, on the other hand, depend on the color of the light source for reflection.

[0176] Optionally, metallicity information can be determined by integrating at least one of the reflectivity, specular and color contrast characteristics, and brightness difference characteristics of the model surface color image.

[0177] Regarding reflectivity, the metallicity of each part of the model surface can be determined by analyzing the color differences in the highlight areas of the model surface color image. For example, if the highlight color is close to the surface color, it indicates that the surface of that part of the model may be metallic; if the highlight color is similar to the color of the light source, it indicates that the surface of that part of the model may be non-metallic.

[0178] Regarding the characteristics of highlights and color contrast, observe the highlight parts in the color image of the model surface. The surface of the metal model shows strong specular reflection (that is, the brightness of the highlight area is high and the color is not affected by the light), while the surface of the non-metallic model usually has weak specular reflection and the highlight color is affected by the light source.

[0179] Regarding the brightness difference characteristics, if the brightness of the highlight is close to the brightness of the model surface itself, it may be metal; if the brightness difference is large, it may be non-metal, etc.

[0180] In a schematic way, metallicity information is output by combining at least one of the above characteristics. For example, the metallicity value corresponding to each pixel is obtained by weight analysis corresponding to at least one of the above characteristics, thereby obtaining metallicity information corresponding to the model surface color image.

[0181] In some embodiments, roughness describes the degree of roughness of the surface microstructure, which affects the degree of light scattering. A rough surface will cause more diffuse reflection, resulting in blurred details and large brightness variations in the image.

[0182] Indicatively, the model surface color image includes detailed information about the model surface (such as small textures, rough areas, etc.) and information about highlight areas; at least one of the highlight and brightness variation characteristics, local brightness variation characteristics, and frequency characteristics represented by the model surface color image is used to determine roughness information.

[0183] Regarding the characteristics of highlight and brightness variations, roughness is closely related to surface reflectivity. Smoother surfaces concentrate light reflection, resulting in strong highlights, while rough surfaces cause light scattering, blurring highlight areas. Observe the highlight distribution in the model surface color image. If the highlights in the model surface color image are sharp and concentrated, it indicates a smoother surface with lower roughness. Conversely, if the highlight areas in the model surface color image are scattered and blurred, it indicates a rougher surface with higher roughness.

[0184] Regarding the characteristics of local brightness variation, the brightness variation is more drastic on rough surfaces because light is scattered at small bumps and depressions on the surface. By analyzing the changes in local brightness, the roughness can be inferred. In addition, local contrast (i.e. the brightness difference between adjacent pixels) can also be used as an indicator of roughness. If the contrast is high, it indicates that the surface is rougher.

[0185] Regarding frequency characteristics, roughness can also be inferred through texture frequency analysis of the model surface color image. For example, rough parts of the model surface usually show more details and higher frequency textures. Alternatively, Fourier transform or wavelet transform can be used to analyze the frequency distribution of the model surface color image. Rough surfaces usually have more components in the high-frequency part.

[0186] In a schematic way, roughness information is output by combining at least one of the above characteristics, such as by using the weight analysis corresponding to each of the above at least one characteristic to obtain the roughness value corresponding to each pixel, thereby obtaining the roughness information corresponding to the model surface color image, etc.

[0187] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0188] In an optional embodiment, image feature representations corresponding to at least two model surface color images are obtained.

[0189] Among them, at least two model surface color images correspond one-to-one with at least two projection viewpoints; illustratively, a feature extraction network is used to perform feature extraction processes on at least two model surface color images respectively to obtain image feature representations corresponding to at least two model surface color images respectively.

[0190] In an optional embodiment, at least two image feature representations are fused to obtain an image fused feature representation.

[0191] Indicatively, image feature representations corresponding to at least two model surface color images are spliced ​​together to obtain image fusion feature representations.

[0192] In an optional embodiment, model surface rendering information corresponding to at least two projected geometry maps is obtained based on image fusion feature representation.

[0193] Optionally, a rendering information analysis model is pre-trained. The rendering information analysis model is a neural network model that includes at least one output. Each output of the rendering information analysis model corresponds to a physical attribute (such as albedo, metallicity, roughness, etc.). The image fusion feature representation is input into the rendering information analysis model to analyze the image fusion feature representation from at least one physical attribute dimension, thereby outputting at least one physical attribute as model surface rendering information. Each projected geometry corresponds to one model surface rendering information, and one model surface rendering information includes at least one of albedo information, metallicity information, and roughness information.

[0194] By extracting color image features from the model surface, an image feature representation with reduced information dimensionality and highlighted key information is obtained. This image feature representation is then fused to obtain a more coherent image fusion feature representation across multiple projection viewpoints. Model surface rendering information is generated based on this image fusion feature representation, ensuring a certain correlation between the model surface rendering information corresponding to different projection viewpoints. This results in a smoother and more harmonious display effect after rendering the model surface from different projection viewpoints.

[0195] In some embodiments, an image conversion diffusion model and a second noise image are obtained.

[0196] Among them, the image transformation diffusion model is a pre-trained model used to generate rendering information for the model; it can also be called the image transformation stable diffusion model.

[0197] Indicatively, the image conversion diffusion model is a pre-trained stable diffusion model that obtains rendering information of the model surface from the model surface color image, belonging to the rendering information analysis model mentioned above. Since the image conversion diffusion model belongs to the diffusion model, the image conversion diffusion model also involves the denoising process in its application.

[0198] Optionally, the second noise image is typically implemented as a random noise image in the application of the image transformation diffusion model; alternatively, the second noise image can also be a noise image obtained by adding noise to the model surface color image, etc., without limitation here. The purpose of obtaining the second noise image is to perform denoising processing on the second noise image through the image transformation diffusion model.

[0199] In some embodiments, the image fusion features are represented as the second denoising condition, and the second noisy image is denoised by an image transformation diffusion model to obtain model surface rendering information corresponding to at least two projected geometric figures.

[0200] Schematic, the second denoising condition is the denoising condition used by the image transformation diffusion model when performing denoising processing on the second noisy image. Guided by this second denoising condition, the denoising process can be biased. Since the image fusion feature representation contains the representation information of the model surface color images corresponding to at least two projection viewpoints, and the model surface color images are obtained under the guidance of rendering information, using the image fusion feature representation as the second denoising condition makes the denoising process tend to obtain information that follows the first display state, preserves the shape of the model geometry, and can restore the representation style of the model surface color image under the corresponding lighting. Furthermore, this process also better reflects the correlation between multiple projection viewpoints. Using the image fusion feature representation as the denoising condition achieves the goal of analytically determining the model surface rendering information while preserving the correlation between various projection viewpoints, thus improving the accuracy of the model surface rendering information. That is, based on the denoising process, model surface rendering information corresponding to at least two projection geometry images is obtained.

[0201] As an illustration, since the image feature representation is obtained by comprehensively analyzing the image feature representations corresponding to at least two projection viewpoints, the image fusion feature representation covers the performance information corresponding to at least two projection viewpoints under the lighting conditions indicated by the rendering guidance information (such as uniform lighting). For example, the image fusion feature representation includes the second feature representations corresponding to at least two projection viewpoints, and each second feature representation represents a projection viewpoint.

[0202] Optionally, the second noisy image is a two-dimensional image. In the process of using the image fusion feature representation as the second denoising condition, each second feature representation in the image fusion feature representation is used as the second denoising condition under the corresponding projection view. Therefore, model surface rendering information corresponding to at least two projection view can be obtained based on the denoising process.

[0203] Indicatively, for the second noise image (if the second noise image is a separate noise image, then the second noise image can be reused; or, second noise images corresponding to at least two projection viewpoints are obtained, etc., each second noise image can be the same random noise image, etc.), each second feature representation in the image fusion feature representation is used as a second denoising condition, and denoising processing is performed on the second noise image to obtain model surface rendering information corresponding to at least two projection viewpoints.

[0204] For example: Based on the number of viewpoints of at least two selected projection viewpoints, obtain a corresponding number of random noise images (which can be the same or different) as the second noise image. For example, obtain 4 random noise images based on the number of viewpoints 4 as the second noise image (or obtain 6 random noise images based on the number of viewpoints 6, which is not limited here). For the second noise image N2, use the second feature representation corresponding to projection viewpoint 1 in the image fusion feature representation as the second denoising condition, and perform denoising processing on the second noise image N1 to obtain the model surface rendering information corresponding to projection viewpoint 1 (such as including the albedo map a1, roughness map r1 and metallicity map m1 corresponding to projection viewpoint 1); similarly, the model surface rendering information corresponding to at least two projection viewpoints can be obtained respectively.

[0205] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0206] like Figure 4 As shown, the following explanation will be given using the multi-view diffusion model 410 and the image-to-image stable diffusion model 420 to implement the above steps 310 to 320 as an example.

[0207] Indicatively, the first three-dimensional model 400 can also be referred to as a three-dimensional mask image, or simply a mesh; multi-view rendering is performed on the first three-dimensional model 400, which is the process of projecting the first three-dimensional model onto at least two projection viewpoints. Therefore, based on the multi-view rendering, projection geometry corresponding to at least two projection viewpoints is obtained.

[0208] Each projection viewpoint corresponds to a projection geometry map, which includes at least one of a 3D coordinate map (xyz map), a depth map, and a normal map. These three maps carry different types of geometric information. The 3D coordinate map represents the 3D coordinates (x, y, z) of each pixel in the first 3D model, and is typically used for the model surface. The depth map represents the distance from the virtual camera to each pixel, typically the depth value of each pixel, and is used to represent the distance information of the first 3D model relative to the virtual camera. The normal map represents the normal vector of each pixel, and is typically used to describe the orientation information of the model surface. The normal map is crucial for lighting and shadow calculations.

[0209] Different projection viewpoints correspond to different virtual cameras. The 3D coordinate map is determined based on the camera coordinate system corresponding to the virtual camera. The depth map is also closely related to the position and observation angle of the virtual camera. The normal map represents the orientation of the model surface and is also closely related to the position and observation angle of the virtual camera. Therefore, corresponding projection geometry can be obtained based on different projection viewpoints. Usually, for a complex first 3D model, the projection geometry corresponding to different projection viewpoints is mostly different, which is not limited here.

[0210] like Figure 4 As shown, this is the content of the six projected geometric figures corresponding to the first 3D model 400 obtained by selecting six projection viewpoints. Figure 4 The six projection angles shown include the frontal angle, rear angle, left-side angle, right-side angle, top-down angle, and bottom-up angle. These are just illustrative examples. You can choose fewer angles (such as four or three projection angles), more projection angles, or projection angles in more observation directions. There are no limitations here.

[0211] In some embodiments, for the multi-view diffusion model 410, the input includes at least two projected geometries corresponding to the first 3D model 400 (e.g., ...). Figure 4 The multi-view diffusion model 410 integrates the projection geometry maps corresponding to the six projection viewpoints and rendering guidance information. The purpose of the multi-view diffusion model 410 is to fuse the rendering guidance information and the projection geometry maps obtained from at least two projection viewpoints, such as obtaining a first fusion feature representation based on the fusion process. The output of the multi-view diffusion model 410 is at least two model surface color images (rbg images) obtained after performing denoising processing on the first noisy image using the first fusion feature representation as the first denoising condition. Figure 4 The image shows the color images of the model surface corresponding to the six projection viewpoints.

[0212] In some embodiments, for the image transformation diffusion model 420, the input includes at least two model surface color images (such as...). Figure 4 The image conversion diffusion model 420 (which uses color images of the model surfaces corresponding to six projection viewpoints) aims to obtain the rendering information of the model surfaces corresponding to each projection viewpoint from these color images. Figure 4 The six projection viewpoints shown correspond to albedo maps (albedo information), metallicity maps (metallicity information), and roughness maps (roughness information) as model surface rendering information for the projection viewpoints.

[0213] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0214] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0215] In this embodiment, a method is described to analyze a first 3D model from at least two projection perspectives, thereby fusing the projection geometry and rendering guidance information corresponding to each of the at least two projection perspectives to obtain the model surface rendering information corresponding to each of the at least two projection perspectives. By comprehensively analyzing the first 3D model from multiple projection perspectives, model surface rendering information with viewpoint correlation can be obtained under the constraint of rendering guidance information, improving the accuracy of the model surface rendering information. Specifically, multi-view diffusion models and image conversion diffusion models are used as methods to obtain model surface color images and model surface rendering information. A progressive denoising process makes the acquisition of model surface rendering information more stable, and the generated model surface rendering information, such as albedo maps, roughness maps, and metallic maps, has high quality and resolution. Furthermore, the progressive denoising process also makes the generated results more detailed and realistic.

[0216] In an optional embodiment, when rendering the surface of the first 3D model using model face rendering information, the vertex coordinates corresponding to the model vertices of the first 3D model are analyzed to achieve the baking rendering purpose through UV coordinate mapping. (Illustrative example follows.) Figure 5 As shown above, Figure 2 The illustrated embodiment can also be implemented as follows: steps 510 to 570; wherein Figure 2 The step 240 shown can also be implemented as steps 540 to 570.

[0217] Step 510: Obtain the first 3D model and the rendering guidance information for the first 3D model.

[0218] Optionally, the first 3D model is a pre-defined 3D model that needs to be baked for rendering. Illustratively, certain rendering effects (such as lighting, shadows, reflections, etc.) can be pre-computed and saved to the texture map; this process is called "baking." The goal of baking rendering is to improve rendering efficiency, especially in scenes requiring real-time rendering (such as game scenes and virtual reality scenes), making the rendering process smoother and more efficient.

[0219] Optionally, the rendering guidance information is used to indicate that the appearance of the first 3D model should be adjusted to a first display state.

[0220] In illustrative terms, the rendering guidance information is information specific to the first 3D model, used to guide the appearance of the first 3D model to be adjusted to the first display state expressed by the rendering guidance information.

[0221] In an optional embodiment, a first 3D model and rendered text are obtained.

[0222] The rendering text is used to describe in text form the adjustment of the display state of the first three-dimensional model. The rendering guidance information includes the rendering text, such as the rendering text used to describe in text form the display state of the first display style as the appearance of the first three-dimensional model after adjustment.

[0223] For illustrative purposes, the rendered text is the text content describing the first 3D model. For example, if the first 3D model is "virtual axe" and the rendered text is "a colored axe", then the first display state is "colored", etc.

[0224] Optionally, the rendered text is text content typed by an object through a text input operation; or, the rendered text is text content selected by an object from preset text content through a text selection operation, etc., without limitation. Optionally, the rendered text is used as rendering guidance information to participate in the fusion process with the projected geometry obtained by projecting from at least two projection perspectives, such as obtaining the text feature representation corresponding to the rendered text as the aforementioned information feature representation.

[0225] Optionally, if the rendering text does not specify a lighting environment, the default omnidirectional uniform lighting environment can be used as the lighting environment corresponding to the rendering guidance information; or, if the rendering text specifies a lighting environment, the lighting environment specified in the rendering text can be used as the lighting environment corresponding to the rendering guidance information, such as the rendering text being "a colored axe with light on top", etc., without any limitation here.

[0226] In an optional embodiment, a first 3D model and rendered text are obtained; a third 3D model conforming to the rendered text is obtained, and the rendering guidance information includes the third 3D model.

[0227] To illustrate, after obtaining the first 3D model and rendered text, a third 3D model is generated based on the rendered text using a pre-trained image generation model. Since the rendered text describes the content of the first 3D model, the third 3D model generated by the image generation model is similar in type to the first 3D model (e.g., both are virtual axes; or both are virtual houses, etc.). However, the third 3D model conforms to the first display state, while the first 3D model is a style other than the first display state. In this case, the third 3D model generated by the image generation model is used as a style reference; even if its actual shape differs from the first 3D model, it will not participate in the subsequent analysis process.

[0228] Optionally, the image generation model includes at least one of the following: Text-to-Mesh (Text2Mesh), TetraNet (TetraNet), and Deep Signed Distance Function (DeepSDF). The Text-to-Mesh model directly generates a 3D mesh from rendered text to obtain a third 3D model; that is, the 3D mesh generated through natural language descriptions can accurately capture the details and features described in the rendered text. The TetraNet model is a neural network-based generative model used to transform rendered text into a mesh model of the third 3D model. Reinforcement learning or variational methods are typically used to optimize the generation process. The Deep Signed Distance Function utilizes neural networks to generate implicit functions for 3D shapes, used to generate the corresponding implicit function from a piece of rendered text, and then generate the third 3D model based on this implicit function.

[0229] Rendering guidance information can be implemented in the form of rendered text or other forms of 3D models used for guidance. Rendering guidance information guides the first 3D model to change its current display state and display other display states. Diverse rendering guidance information can avoid the limitations of the rendering scene, improve the rendering detail, better handle complex rendering tasks, and balance realism with computational efficiency.

[0230] In some embodiments, a 3D model material library is obtained.

[0231] The 3D model library includes at least one styled 3D model labeled with a style tag. For illustration purposes, the style tag is a text description, and each styled 3D model is labeled with a style tag that matches its model attributes (such as shape, style, etc.).

[0232] In some embodiments, a style 3D model matching the rendered text is searched from a 3D model material library based on style tags as a third 3D model.

[0233] This is illustrative of comparing rendered text with style tags corresponding to style 3D models in a 3D model asset library, and selecting the style 3D model corresponding to the style tag that matches the rendered text as the third 3D model, etc., without limitation.

[0234] In an optional embodiment, a first three-dimensional model and a fourth three-dimensional model are obtained. The fourth three-dimensional model is a three-dimensional model displayed in a first display state, which is a display state after guiding the adjustment of the first three-dimensional model. The rendering guidance information includes the fourth three-dimensional model.

[0235] In illustrative terms, the style of the first 3D model can be adjusted not only through text-based rendering but also through a 3D model in model form; for example, the fourth 3D model is displayed in the first display state with rendering guidance information.

[0236] Optionally, the model style of the fourth 3D model is the first display state, but its model shape can be the same as or different from the model shape of the first 3D model, which is not limited here.

[0237] In an optional embodiment, a fifth three-dimensional model representing a first display state is obtained, the fifth three-dimensional model including at least one of a third three-dimensional model and a fourth three-dimensional model.

[0238] As an illustration, the third 3D model obtained from the rendering text can be used as the fifth 3D model, or the fourth 3D model obtained directly can be used as the fifth 3D model.

[0239] In an optional embodiment, the fifth 3D model is subjected to delighting processing to obtain a sixth 3D model corresponding to the fifth 3D model as rendering guidance information.

[0240] Among them, the sixth three-dimensional model is used to characterize the surface material properties when the first display state is presented after removing the influence of other lighting besides ambient light.

[0241] Optionally, a pre-trained image delighting model can be used to perform delighting processing on the fifth 3D model to remove other lighting effects besides uniform ambient light, such as removing the strong light effects of strong light sources (e.g., simulated sunlight, simulated spotlights, etc.), and also removing reflected light, highlights, shadows, etc., without limitation here.

[0242] Therefore, compared to the fifth 3D model, which also represents the first display state, the sixth 3D model is a more uniform representation of the first display state after removing the influence of lighting. This significantly reduces lighting interference and improves the physical accuracy of subsequent rendering. Furthermore, it can be better adapted to other lighting conditions, avoiding the constraints of a single lighting mode and enhancing rendering realism and accuracy. Therefore, the sixth 3D model can also be referred to as an omnidirectional uniform lighting result.

[0243] In some embodiments, the illumination distribution information corresponding to the fifth three-dimensional model is obtained.

[0244] The illumination distribution information includes at least one of ambient illumination information, direct illumination information, and indirect illumination information.

[0245] In illustration, ambient lighting information is globally uniform illumination within a 3D scene, typically representing light sources from all directions, similar to scattered light in the atmosphere. Therefore, ambient lighting information lacks directionality and intensity variation, providing a uniform base brightness for 3D models within the scene. Ambient lighting generally does not produce shadows or highlights, balancing the brightness within the 3D scene so that unseen areas do not become completely black, maintaining overall lighting balance.

[0246] Direct lighting is illumination where a specific light source (such as a point light, spotlight, or directional light) directly illuminates the surface of a 3D model, producing variations in brightness. These rays travel directly from the light source to the model's surface, typically exhibiting both directionality and intensity variations. Direct lighting results in noticeable highlights, shadows, and variations in light intensity across the object's surface. It is the most intuitive and controllable type of lighting and is commonly used to highlight object details and contours.

[0247] Indirect lighting refers to the illumination produced when light reflects off the surface of a 3D model in a 3D scene and then shines onto the surface of other 3D models. Indirect lighting is typically caused by indirect reflections from ambient light sources, thus exhibiting some scattering and blurring. It is usually less intense and can affect color (such as color shift), similar to how it fills in shadows and enhances lighting details in the real world. Indirect lighting makes the appearance of 3D models within a 3D scene appear more natural, such as light reflecting off walls and then onto other 3D models.

[0248] In some embodiments, the fifth 3D model is subjected to delighting processing based on the illumination distribution information to obtain the sixth 3D model as rendering guidance information.

[0249] Optionally, the direct and indirect lighting information corresponding to the fifth 3D model is removed to obtain a sixth 3D model that retains the ambient lighting information, which is then used as rendering guidance information.

[0250] In illustrative terms, when ambient lighting information, direct lighting information, and indirect lighting information are determined as lighting distribution information based on the lighting model, considering that the ambient lighting information itself is uniform, it usually does not produce obvious lighting differences or shadow effects.

[0251] Therefore, during the delighting process, ambient lighting information is preserved to maintain the overall balance and naturalness of the 3D model, while direct and indirect lighting information is removed. This results in a fifth 3D model that retains ambient lighting information, which serves as the sixth 3D model. This significantly improves the accuracy of subsequent rendering processes based on rendering guidance information, effectively addressing the material reconstruction needs in complex lighting scenes.

[0252] In a schematic manner, with the sixth 3D model as the rendering guidance information, the model feature representation corresponding to the sixth 3D model is obtained as the information feature representation of the fusion with the projection geometry obtained by at least two projection viewpoints.

[0253] Step 520: Project the first 3D model to obtain at least two projected geometric figures.

[0254] Among them, the i-th projection geometry in at least two projection geometry diagrams is used to describe the geometry obtained by projecting the first three-dimensional model under the i-th projection viewpoint, where i is a positive integer; the at least two projection geometry diagrams correspond to different projection viewpoints.

[0255] In an optional embodiment, during the process of projecting the first three-dimensional model onto the view space corresponding to at least two projection perspectives, pixel information corresponding to at least two model pixels in the first three-dimensional model is obtained.

[0256] The pixel information includes at least one of the following: pixel position, pixel normal, and pixel depth.

[0257] Indicatively, the view space describes the three-dimensional space observed by the virtual correlation corresponding to the projection viewpoint. Each view corresponds to a view space coordinate system, and pixel information represents the state of the model pixels in the corresponding view space coordinate system.

[0258] Pixel position refers to the corresponding location of a model pixel in a 2D image; pixel normal is the outward perpendicular direction of a model pixel in 3D space (i.e., the unit vector of the surface facing outward), used to calculate the interaction between lighting and the surface, determining the lighting intensity and surface reflection effect. Pixel depth refers to the distance of a model pixel relative to the viewpoint of the virtual camera, usually expressed as a depth value. The smaller the depth value, the closer it is to the virtual camera; the larger the depth value, the farther away it is.

[0259] In an optional embodiment, projection geometry corresponding to at least two projection viewpoints is obtained by rendering pixel information.

[0260] The projected geometry map includes at least one of three types: a 3D coordinate map, a depth map, and a normal map. These three maps each carry different types of geometric information. The 3D coordinate map represents the 3D coordinates (x, y, z) of each pixel in the first 3D model and is typically used for the model surface of the first 3D model. The depth map represents the distance from the virtual camera to each pixel, typically the depth value of each pixel, and is used to represent the distance information of the first 3D model relative to the virtual camera. The normal map represents the normal vector of each pixel and is typically used to describe the orientation information of the model surface.

[0261] By projecting the first 3D model onto the view space corresponding to the projection perspective, pixel information of the model's pixels can be obtained, and then projection geometry maps corresponding to different projection perspectives can be obtained from different view spaces. The 3D coordinate map, depth map, and normal map, which serve as projection geometry maps, are crucial for lighting and shadow calculations, helping to obtain more targeted and accurate geometric information under different projection perspectives, thereby significantly improving the accuracy of subsequent 3D rendering based on the projection geometry maps.

[0262] Step 530: Fuse the rendering guidance information and at least two projected geometry maps to generate model surface rendering information corresponding to at least two projected geometry maps respectively.

[0263] Among them, the model surface rendering information is used to render the display state of the model surface of the first 3D model under the projection view.

[0264] Indicatively, using the first display state expressed by the rendering guidance information as a style reference and the projected geometry obtained from at least two projection viewpoints as shape references, model surface rendering information containing multi-view correlation information and conforming to the first display state is generated. For example, if at least two projection viewpoints correspond to projected geometry, and each projected geometry corresponds to model surface rendering information, then the resulting information is model surface rendering information corresponding to at least two projection viewpoints.

[0265] Step 540: Obtain the vertex coordinates corresponding to at least one model vertex in the first 3D model.

[0266] Among them, vertex coordinates are the position information of the model vertex in the three-dimensional space of the first three-dimensional model.

[0267] Indicatively, vertex coordinates in three-dimensional space are usually represented by (x, y, z), where x is the coordinate of the model vertex on the X-axis; y is the coordinate of the model vertex on the Y-axis; and z is the coordinate of the model vertex on the Z-axis.

[0268] Optionally, the vertex coordinates of at least one model vertex can be obtained from the model file corresponding to the first 3D model; or, the vertex coordinates of at least one model vertex can be obtained by importing the first 3D model into a 3D graphics application.

[0269] Step 550: Obtain the camera optical center coordinates corresponding to the virtual camera.

[0270] Among them, the virtual camera is used to observe three-dimensional space.

[0271] In illustrative terms, a virtual camera is a tool used in computer graphics and computer vision to simulate a real camera. It is typically used to render 3D scenes or to capture and process 3D data. A virtual camera defines a view from which objects in 3D space are observed, and each virtual camera corresponds to a projection viewpoint.

[0272] The camera optical center coordinates, also known as the camera origin, are a very important reference point for the virtual camera. It is the origin of the camera coordinate system corresponding to each projection viewpoint, and can also be understood as the position of the virtual camera. The virtual camera observes and renders the 3D scene from the camera optical center coordinates.

[0273] Step 560: Based on the camera optical center coordinates and vertex coordinates, obtain the viewing direction corresponding to at least one model vertex.

[0274] The line-of-sight direction is used to characterize the direction from which the virtual camera observes the vertices of the model.

[0275] To illustrate, the line of sight is from the camera's optical center toward the model's vertex, so the line of sight can be obtained by calculating the vector between the camera's optical center coordinates and the vertex coordinates.

[0276] For any virtual camera corresponding to a projection viewpoint, determine the vertex coordinates of the model vertex observed from that projection viewpoint; perform vector operations between the camera optical center coordinates and vertex coordinates of the virtual camera to obtain the implementation direction.

[0277] Step 570: Using the viewing direction as a reference, perform texture rendering on the surface of the first 3D model using the model surface rendering information corresponding to at least two projected geometry diagrams to obtain the second 3D model.

[0278] Indicatively, the line-of-sight direction is related to the virtual camera and model vertices. This direction significantly enhances the realism and dynamic interactivity of the rendering, and also helps to better adjust material representation dynamically in conjunction with changes in viewpoint. Therefore, when performing texture rendering on the surface of the first 3D model using model surface rendering information corresponding to at least two projection viewpoints, the line-of-sight direction between the model vertices on the model surface and the virtual camera is used as a reference. Texture rendering is then performed on the observable portion of the model surface using the model surface rendering information corresponding to the projection viewpoints, thus enhancing the realism and lifelikeness of the second 3D model.

[0279] In an optional embodiment, the texture coordinates corresponding to at least two model surface points in the first three-dimensional model are determined by the model unfolding state of the first three-dimensional model.

[0280] The model surface points include model vertices.

[0281] To illustrate, in texture mapping of a 3D model, texture coordinates are typically used to associate the geometry of the 3D model (such as vertices, faces, etc.) with a 2D texture image (such as the 2D model face rendering information corresponding to each projection viewpoint). This process is called UV mapping; it maps the pixels of the texture image to the surface of the 3D model by unfolding the 3D model into a 2D plane.

[0282] Alternatively, the UV mapping process can be briefly described as follows.

[0283] The surface of the 3D model is unfolded onto a 2D plane, so that each point on the model surface corresponds to a certain position on the unfolded 2D plane. Mapping methods such as projection, stretching or segmentation are used to map the coordinates of each model surface point (usually the model vertex) to coordinates in UV space, that is, to obtain the texture coordinates (u, v) of each model surface point. The texture coordinates indicate which part of the texture image corresponds to the model surface point.

[0284] In an optional embodiment, rendering sub-information corresponding to the texture coordinates of at least two model surface points is determined from the model surface rendering information corresponding to at least two projected geometries.

[0285] The nth rendering sub-information is used to render the model surface point corresponding to the nth texture coordinate, where n is a positive integer.

[0286] In a schematic way, each texture coordinate refers to a model surface point on the first 3D model. After finding the rendering sub-information from the model surface rendering information based on the texture coordinates, the model surface point referred to by the corresponding texture coordinate can be rendered through the rendering sub-information to realize the baking rendering process.

[0287] Furthermore, the involvement of texture coordinates in the above process can be omitted. For example, from the model surface rendering information corresponding to at least two projected geometry maps, rendering sub-information corresponding to at least two model surface points can be determined, and the nth rendering sub-information is used to render the nth model surface point. For instance, aligning the unfolded model image after the first 3D model is unfolded with at least one of the albedo map, metallic map, and roughness map representing the model surface rendering information, where the unfolded model image includes multiple model surface points, and determining the rendering sub-information corresponding to each model surface point based on the image alignment result, thus obtaining the rendering sub-information corresponding to multiple model surface points.

[0288] To illustrate, since texture coordinates are two-dimensional information and model surface rendering information is also two-dimensional information, the rendering pixels that are aligned with the texture coordinates can be found from the model surface rendering information (such as pixels in the albedo map, pixels in the roughness map, pixels in the metallicity map, etc.), and the rendering sub-information corresponding to the rendering pixels can be used as the information corresponding to the texture coordinates.

[0289] Among them, the rendering sub-information is used to render the model surface points corresponding to the texture coordinates.

[0290] In illustrative purposes, the rendering sub-information includes at least one of several types of information such as albedo value, roughness value, and metallicity value, without limitation here.

[0291] In an optional embodiment, with the viewing direction as a reference, and based on the correspondence between texture coordinates and rendering sub-information, the model surface points corresponding to the texture coordinates are rendered using the rendering sub-information to obtain the second three-dimensional model.

[0292] In illustration, texture coordinates are two-dimensional coordinates determined after unfolding the first 3D model. Based on the correspondence between texture coordinates and rendering sub-information, the rendering sub-information is specifically baked and rendered onto the model surface points indicated by the texture coordinates. This quantifies the rendering process and enhances the dynamic realism and display accuracy, balancing efficiency and visual effects. Optionally, when determining the rendering sub-information corresponding to each model vertex to achieve baking rendering, interpolation can also be used to render more detailed model surface points in the middle; this is not limited here.

[0293] In some embodiments, the first texture coordinates corresponding to the first model surface points are obtained; when at least two first rendering sub-information corresponding to the first texture coordinates are determined from at least two model surface rendering information, the rendering weight corresponding to the viewing direction is obtained; the at least two first rendering sub-information are adjusted by the rendering weight to obtain the rendering sub-information corresponding to the first texture coordinates.

[0294] Among them, the rendering weight is used to balance the rendering contribution of at least two first rendering sub-information to the first model surface point corresponding to the first texture coordinate.

[0295] In illustrative terms, the purpose of obtaining rendering weights (or baking weights) is that at least one model vertex may lie on the model surface corresponding to multiple virtual cameras (multiple projection views). In this case, rendering weights are needed to balance the rendering information of the model surfaces corresponding to different projection views, thereby improving the accuracy and consistency of image rendering and ensuring the coordinated effect of multi-view rendering. Therefore, introducing rendering weights corresponding to the viewing direction to balance different rendering sub-information corresponding to the same texture coordinate can effectively avoid rendering errors and improve the realism of the rendering. For example, for each projection view, the front view generally has a larger weight, resulting in a more believable result.

[0296] Therefore, rendering weights can be used to adjust multiple first rendering sub-information that may correspond to a first texture coordinate, so as to obtain a unique rendering sub-information corresponding to the first texture coordinate, avoiding the problem of repeated baking rendering that affects the rendering effect, and making the second 3D model more vivid and realistic.

[0297] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0298] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0299] In some embodiments, such as Figure 6 As shown, the process of integrating the image delighting model 610, the multi-view diffusion model 620, and the image transformation diffusion model 630 to obtain model surface rendering information for baking rendering of the first three-dimensional model and obtaining the second three-dimensional model is achieved.

[0300] like Figure 6As shown, the input is either rendered text or a fourth 3D model; if rendered text is input, it is converted into a third 3D model, and then analysis is performed based on the third 3D model; if a fourth 3D model is input, subsequent analysis is performed based on the fourth 3D model; in the subsequent analysis process, the third 3D model or the fourth 3D model is collectively referred to as the fifth 3D model.

[0301] Optionally, the fifth 3D model is subjected to delighting processing through the image delighting model 610 to obtain the sixth 3D model as rendering guidance information (i.e., omnidirectional uniform lighting result). The sixth 3D model is a 3D model with a first display state and uniform ambient lighting.

[0302] The multi-view diffusion model 620 analyzes the model geometry of the first 3D model to be baked and rendered under at least two projection views, and analyzes the omnidirectional uniform lighting result output by the image delighting model 610 as rendering guidance information for the first display state. Illustratively, when performing denoising processing on the first noisy image, the multi-view diffusion model 620 combines the omnidirectional uniform lighting result and the model geometry corresponding to at least two projection views as denoising control information (first denoising condition), thereby outputting a model surface color image (RGB image) by the multi-view diffusion model 620.

[0303] The model surface color images corresponding to at least two projection viewpoints are analyzed using an image transformation diffusion model 630 to obtain model surface rendering information (such as albedo maps, metallic maps, and roughness maps); related content can be found in [reference needed]. Figure 4 Note that no restrictions are imposed here.

[0304] In an optional embodiment, refer to as follows Figure 6 The content shown first explains the overall processing method for stages one through three.

[0305] To illustrate, the method of generating multiple viewpoint images (model surface color images corresponding to multiple projection viewpoints) at once is used to cover as much of the visible area of ​​the model as possible, such as selecting 4 projection viewpoints or 6 projection viewpoints; in this process, the multi-viewpoint consistency is ensured by the multi-viewpoint self-attention mechanism of the Stable Diffusion (SD) model.

[0306] Optionally, when selecting multiple viewpoints, the Janus face problem can be avoided by selecting a priori viewpoint (observing the front) and a rear viewpoint (observing the back). The Janus face problem usually refers to the duality of two-dimensional images or three-dimensional models. For example, a three-dimensional model may exhibit one property and another property in certain situations. Avoiding the Janus face problem can effectively overcome potential problems in model surface processing, lighting rendering, etc.

[0307] Then, the multi-view images are converted into corresponding physically based rendering information, namely the model surface rendering information mentioned above, including at least one of albedo texture, metallic texture and roughness texture.

[0308] Finally, the albedo map, metallic map, and roughness map are baked back into the first 3D model (mesh) to be rendered, thereby generating the final second 3D model with PBR texture.

[0309] In addition, such as Figure 6 As shown, the model training process is further explained below; including the training process of the multi-view diffusion model 620 in stage one and the training process of the image conversion diffusion model 630 in stage two.

[0310] (I) Phase One

[0311] Indicatively, if input rendering text is received, it is first converted into an image (third 3D model). For the image converted from text (third 3D model) or the directly input fourth 3D model (the third and fourth 3D models can be collectively referred to as the fifth 3D model), the lighting effects of the image can be reduced through a single-image delighting model, converting it into an image with omnidirectional uniform lighting (i.e., the sixth 3D model). The purpose of this process is to eliminate the influence of lighting on multi-view generation, ensuring that the subsequently generated multi-view images have omnidirectional uniform lighting effects, i.e., the images have no obvious shadows.

[0312] Alternatively, many works, including Text to LaTeX (Text2Tex), use control networks, but they are all for generating single images (single images). For white film coloring tasks, even if only single image generation with 4 projection views is allowed, it will cause obvious multi-view inconsistency problems.

[0313] In this embodiment, the control network in SD is generalized. As shown in Formula 3 below, in addition to using the omnidirectional uniform light image (sixth 3D model) as the text control signal c in traditional SD... t In addition, the model geometry (such as 3D coordinate map, depth map, normal map, etc.) is obtained after feature extraction to obtain c. f As an additional control signal, analysis and processing are achieved through cross-attention as shown in the above-mentioned figure.

[0314] Formula 3:

[0315]

[0316] Formula 3 is based on the c of the sixth three-dimensional model. t and the c of the model geometry f The loss function for training the multi-view diffusion model 620 with control signals; where, It is the first loss value used for training the multi-view diffusion model 620. During the training process, it is used to measure the difference between the predicted color image output by the multi-view diffusion model 620 and the label color image corresponding to the sample geometry (e.g., each model geometry corresponds to a label color image of the current projection view). The purpose is to minimize the first loss value. z0 represents the expected value (or average value); z0 is the noisy image that needs to be denoised (such as the first noisy image); t is the time step, which represents a time or step marker in the diffusion model. This indicates that it comes from the standard normal distribution. Real noise is used to simulate random perturbations introduced into the image; ∈ θ (z0,t,c t ,c f () represents the multi-view diffusion model 620 at a given time step t, condition c t And condition c f The noise predicted based on the input image z0 aims to be as close as possible to the true noise ∈, i.e., to minimize the first loss value. The square of the L2 norm (Euclidean distance), also known as the squared error, is used to calculate the noise in the prediction. θ (z0,t,c t ,c f The difference between () and the true noise ∈ is squared.

[0317] The aforementioned objective is for the multi-view diffusion model 620 to learn how to recover the original image from a noisy image.

[0318] This process involves rendering model geometry (such as depth maps) from multiple projection perspectives (e.g., front, back, left, right, bottom, and top) simultaneously, and then stitching these perspectives together for denoising at the SD latent feature dimension. Furthermore, to maximize the reuse of the original control network weights, each model geometry undergoes separate feature extraction via the control network and is applied to its corresponding location. This allows for the simultaneous generation of model surface color images (RGB images) corresponding to multiple projection perspectives. Since multiple images are stitched together at the feature dimension, they can share computation in self-attention analysis, ensuring consistency across multiple perspectives and avoiding the Janus Face problem.

[0319] Based on the above process, the generated RGB image not only conforms to the text prompt (constrained by the omnidirectional uniform light image) but also conforms to the given control signals (the shapes of the model geometry corresponding to multiple projection viewpoints). For example... Figure 7 As shown, the shapes and depths of the RGB images 710 and the model geometry 720 corresponding to the multiple projection viewpoints are consistent. For example, the edges of the RGB images match the edges of the depth maps, and the image details also match the trend of depth changes.

[0320] (II) Phase Two

[0321] like Figure 6 As shown in Phase 2, this scheme extends SDimg2img based on the traditional image (img2img) diffusion model. Considering that the traditional SDimg2img supports a single RGB image as input, if a single image is used for conversion during the PBR material generation process, the converted multi-view PBR material is prone to inconsistent perspectives.

[0322] In this embodiment of the application, based on the model geometry from multiple projection views obtained in the first stage above, model surface color images (RGB images) corresponding to multiple projection views are obtained and constrained by rendering guidance information. The RGB images from multiple views are used as input to the improved image transformation diffusion model 630. Each RGB image is converted into a latent feature representation and stitched together to generate an overall latent (such as the image fusion feature representation mentioned above).

[0323] The latent feature representations corresponding to each RGB image can also be fused with image information from other projection perspectives through self-attention analysis, thereby ensuring better image consistency; so as to output multi-view PBR material images based on image fusion feature representations.

[0324] During the training of the image conversion diffusion model 630, in order to ensure the accuracy of the generation of each material in PBR, the predicted rendering information (predicted PBR) can be rendered onto the corresponding input image (such as the 3D model of the sample to be rendered during training), and the predicted RGB images corresponding to multiple projection views are obtained respectively. Then, the rendering loss between the predicted RGB image and the sample RGB image under the same projection view is calculated. The sample RGB image is obtained by rendering the sample 3D model with the label rendering information (predicted PBR) during training. The projection views of multiple sample RGB images and multiple predicted RGB images correspond one-to-one.

[0325] Alternatively, the rendering loss can be calculated using at least one of the following functions: L1 loss function, Perception loss function, etc.

[0326] Where loss1 = ||rgb_input – rgb_render||; rgb_render represents the predicted RGB image; rgb_input represents the sample RGB image, which is the formula for calculating the L1 loss function; a pre-trained VGG model is used to extract multi-size features from rgb_input and rgb_render respectively, and then the L1 loss function is calculated on features of the same size, which can eliminate the influence of illumination estimation accuracy on image loss constraints.

[0327] In some embodiments, to enhance the stability of PBR materials, different lighting libraries can be pre-set, where the lighting library light = [1, ..., N], and N represents N types of lighting. Then, the predicted rendering information (such as albedo / metallic / roughness) is rendered under different lighting conditions using these multiple lighting options to obtain the corresponding rgb_render_light images (i.e., the predicted RGB images corresponding to different lighting conditions). For the sample rendering information corresponding to the sample RGB image, the corresponding rgb_gt_light image can also be rendered using the same lighting (i.e., the sample RGB images corresponding to different lighting conditions). Various loss functions can be applied to the render image and the corresponding ground truth image rendered under the same lighting conditions, such as the L1 loss function: loss_1ight = ||rgb_render_light – rgb_gt_light||, etc., which are not limited here.

[0328] (III) Phase Three

[0329] Optionally, normals are calculated for the first 3D model to be rendered to obtain the normal n of each model vertex i. iEach projection viewpoint obtains its rendering weight according to the pose of the virtual camera, as shown in Formula 4 below.

[0330] Formula 4:

[0331]

[0332] Where c is the optical center position of the virtual camera; p i These are the vertex coordinates of the model's vertices; the viewing direction (p) is obtained from the camera's optical center position and the vertex coordinates. i -c)*n i And obtain the absolute value of the cosine value of the normal (|p i -c|*n)|n i |;w i This represents the rendering weight; a larger value indicates a closer proximity to verticality, meaning a more positive and believable orientation. Furthermore, w is strengthened by exponentiation through exp. i Weighted discrimination.

[0333] The purpose of obtaining rendering weights (or baking weights) is that at least one model vertex may lie on the model surface corresponding to multiple virtual cameras (multiple projection views). In this case, rendering weights are needed to balance the rendering information of the model surfaces corresponding to different projection views, so as to improve the accuracy and consistency of image rendering and ensure the coordinated effect of multi-view rendering. For example, for each projection view, the front face generally has a larger weight, and the generated result is more reliable.

[0334] In some embodiments, the process of optimizing baking is described as follows.

[0335] In a schematic example, the optimization objective X is to obtain a high-quality multi-texture map (predictive rendering information), and it is randomly initialized. The predicted 3D model obtained after rendering the sample 3D model using sample RGB images corresponding to multiple projection views is used as the ground truth map, and the rendering weight w shown in Formula 4 is used. i X is optimized through multiple differentiable rendering iterations, and the optimization formula is shown in Formula 5 below.

[0336] Formula 5:

[0337]

[0338] in, This represents the error between image pairs corresponding to n projection viewpoints. An image pair corresponding to a projection viewpoint includes the actual rendered image of the i-th projection viewpoint. The predicted rendered image obtained from the i-th projection viewpoint can be used to measure the loss difference between the predicted 3D model and the real 3D model, which includes n projection views.

[0339] The predicted rendered image corresponding to the i-th projection viewpoint is represented by the camera pose T of the i-th virtual camera. i and texture Figure X The rendered image (each pixel of the predicted rendered image is derived from the texture based on the camera pose) is obtained. Figure X (Mid-sampled), the camera poses are different for different projection viewpoints i, but they all share the same texture. Figure X .

[0340] The objective is to minimize the loss shown in Formula 5, that is, by adjusting the texture. Figure X (Including albedo maps, roughness maps, and metallic maps) to reduce the error between the predicted rendered image and the real rendered image under each projection viewpoint, that is, to reduce the difference between the baked 3D model under multiple projection views and the 3D model rendered based on the real image, so that the predicted rendered image is as close as possible to the real rendered image, that is, to improve the realism of the 3D model after rendering and baking; after hundreds of iterations (about hundreds of milliseconds), the optimized texture map in the UV space can be obtained.

[0341] Optionally, three texture images will be generated as the final PBR texture information, i.e. model face rendering information, such as albedo texture, metallic texture and roughness texture, etc., which are not limited here.

[0342] In an optional embodiment, such as Figure 8 As shown, based on the above method, the first 3D model 810 to be baked and rendered is obtained as a virtual axe. The rendering text 820 used to guide the rendering and display of the first 3D model in the first display state is "A colorful axe". This can be in Chinese, but could also be in English or other formats; no limitation is made here. Based on the above... Figure 6 The proposed solution can obtain a virtual axe in its first display state (color) based on the baking and rendering process (the color is indicated by thickened lines and partial color descriptions in the figure), which is the second three-dimensional model 830.

[0343] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0344] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0345] In this embodiment, a fifth 3D model of the first display state can be additionally obtained as a style reference to remove the influence of additional lighting information through delighting processing, making the lighting environment of the rendering guidance information more stable, and the first display state more targeted. Compared with directly applying text as a control signal, the control effect is more stable, thereby improving the accuracy of the result after noise reduction processing.

[0346] In one optional embodiment, taking a model rendering program (such as a webpage, software, or mini-program) with a baking rendering function as an example, the appearance of a given first 3D model can be adjusted by inputting rendering text or other methods that specify the first display state.

[0347] Indicative, such as Figure 9 The diagram shows an interface for texture generation to achieve baked rendering. It includes a configuration area 910 and a display area 920. Operations such as model selection and import are performed on the configuration area 910 to determine a first 3D model. A third 3D model for the first display state can also be provided. Furthermore, operations such as text input and text selection can be performed on the configuration area 910 to determine the rendered text describing the first display state.

[0348] At this point, the rendered text representing the first display state and / or the third 3D model can serve as rendering guidance information, or the rendering guidance information can be obtained by removing the lighting from a given 3D model. The rendering guidance information is used to express adjusting the appearance of the first 3D model to the first display state. For example, the first 3D model is a virtual axe.

[0349] When the model rendering program analyzes based on the virtual axe and rendering guidance information (such as the rendering text "a colored axe"), it obtains the projected geometry of the first 3D model projected from at least two projection perspectives.

[0350] Among them, the projection geometry is used to describe the geometric shape of the first 3D model projected under at least two projection viewpoints. For example, the frontal geometry of the virtual axe under the frontal viewpoint and the rearward geometry under the rearward viewpoint.

[0351] In addition, the model rendering program can also generate model surface rendering information corresponding to the projection geometry by fusing the first display state expressed by the rendering guidance information and the projection geometry obtained by at least two projection viewpoints.

[0352] Among them, the model surface rendering information is used to express the texture information when the first display state is rendered on the model surface corresponding to the projected geometry.

[0353] Schematic, by fusing the first display state expressed by the rendering guidance information and the projected geometry obtained from at least two projection viewpoints, model surface color images corresponding to at least two viewpoints are generated, with a one-to-one correspondence between the at least two projection viewpoints and at least two model surface color images. The model surface color images are used to describe the surface color of the model surface under the first lighting conditions corresponding to the rendering guidance information; based on the at least two model surface color images, model surface rendering information corresponding to the projected geometry obtained from the at least two projection viewpoints is generated.

[0354] Finally, the model rendering program renders the surface of the first 3D model using the model surface rendering information to obtain the second 3D model. The second 3D model is a 3D model obtained by adjusting the first 3D model in the first display state.

[0355] In a schematic manner, the vertex coordinates corresponding to at least one model vertex in the first three-dimensional model are obtained. The vertex coordinates are the position information of the model vertex in the three-dimensional space where the first three-dimensional model is located. The optical center coordinates of the camera corresponding to the virtual camera are obtained. The virtual camera is used to observe the three-dimensional space. Based on the camera optical center coordinates and vertex coordinates, the viewing direction corresponding to at least one model vertex is obtained. The viewing direction is used to represent the direction of observation of the model vertex from the virtual camera. With the viewing direction as a reference, the model surface of the first three-dimensional model is textured by using at least two model surface rendering information to obtain the second three-dimensional model.

[0356] like Figure 9 As shown, by rendering text and at least one of the third 3D models, the first 3D model is guided to change to a first display state, thereby displaying the first 3D model in the first display state in the display area 920, that is, displaying the second 3D model (in the figure, the style is changed to color by bolding the lines and some color descriptions).

[0357] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0358] Figure 10 A structural block diagram of a rendering apparatus for a three-dimensional model provided in an exemplary embodiment of this application is shown below. Figure 10 As shown, the device includes the following parts:

[0359] The acquisition module 1010 is used to acquire the first three-dimensional model and the rendering guidance information for the first three-dimensional model;

[0360] The projection module 1020 is also used to project the first three-dimensional model to obtain at least two projected geometric figures, wherein the at least two projected geometric figures correspond to different projection perspectives and respectively refer to different geometric shapes;

[0361] The fusion module 1030 is used to fuse the rendering guidance information and the at least two projected geometry maps respectively to generate model surface rendering information corresponding to the at least two projected geometry maps respectively. The model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projection view.

[0362] The rendering module 1040 is used to render the model surface of the first three-dimensional model using the model surface rendering information corresponding to the at least two projected geometry diagrams, to obtain the second three-dimensional model.

[0363] In an optional embodiment, the fusion module 1030 is further configured to fuse the rendering guidance information and the i-th projection geometry of the at least two projection geometry to generate a model surface color image corresponding to the i-th projection viewpoint; the rendering guidance information corresponds to a first lighting condition, and the model surface color image corresponding to the i-th projection viewpoint is used to describe the surface color of the model surface observed from the i-th projection viewpoint under the first lighting condition, where i is a positive integer; based on the model surface color images corresponding to the at least two projection geometry, the model surface rendering information corresponding to the at least two projection geometry is generated.

[0364] In an optional embodiment, the fusion module 1030 is further configured to obtain the information feature representation corresponding to the rendering guidance information; obtain the geometric feature representation corresponding to the i-th projection geometry; fuse the information feature representation and the geometric feature representation corresponding to the i-th projection geometry to obtain the i-th first feature representation; and generate the model surface color image corresponding to the i-th projection viewpoint based on the i-th first feature representation.

[0365] In an optional embodiment, the fusion module 1030 is further configured to obtain geometric feature representations corresponding to the at least two projected geometric figures respectively; perform feature concatenation on the at least two geometric feature representations to obtain geometric concatenated feature representations; perform cross-attention analysis on the information feature representations and the geometric concatenated feature representations to obtain attention concatenated feature representations, wherein the attention concatenated feature representations include attention sub-feature representations corresponding to the at least two geometric feature representations respectively; perform self-attention analysis on the attention sub-feature representations corresponding to the at least two geometric feature representations respectively to obtain self-attention feature representations corresponding to the at least two attention sub-feature representations respectively, wherein the self-attention feature representations corresponding to the at least two attention sub-feature representations include the self-attention feature representation corresponding to the i-th projection viewpoint, and the self-attention feature representation corresponding to the i-th projection viewpoint serves as the i-th first feature representation corresponding to the i-th projection viewpoint.

[0366] In an optional embodiment, the fusion module 1030 is further configured to perform cross-attention analysis on the geometric feature representation and the information feature representation corresponding to the i-th projected geometry to obtain a first attention feature representation corresponding to the i-th geometric feature representation; obtain geometric feature representations corresponding to the at least two projected geometrys respectively, including the geometric feature representation corresponding to the i-th projected geometry; perform self-attention analysis on the first attention feature representations corresponding to the at least two geometric feature representations respectively to obtain a second attention feature representation corresponding to the at least two first attention feature representations respectively, wherein the second attention feature representation is an enhanced feature representation based on characterizing the at least two first attention feature representations, and the second attention feature representation corresponding to the i-th projection viewpoint serves as the i-th first feature representation corresponding to the i-th projection viewpoint.

[0367] In an optional embodiment, the fusion module 1030 is further configured to acquire a multi-view diffusion model and a first noise image, wherein the multi-view diffusion model is a pre-trained model for acquiring color images corresponding to the projection viewpoints; the first feature is represented as the first denoising condition corresponding to the i-th projection viewpoint, and the first noise image is denoised using the multi-view diffusion model to obtain the model surface color image corresponding to the i-th projection viewpoint.

[0368] In an optional embodiment, the fusion module 1030 is further configured to acquire image feature representations corresponding to at least two model surface color images respectively; fuse the image feature representations corresponding to the at least two model surface color images respectively to obtain image fusion feature representations; and obtain model surface rendering information corresponding to the at least two projected geometry images respectively based on the image fusion feature representations, wherein the model surface rendering information includes at least one of albedo information, metallicity information and roughness information.

[0369] In an optional embodiment, the fusion module 1030 is further configured to acquire an image transformation diffusion model and a second noise image, wherein the image transformation diffusion model is a pre-trained model for generating model rendering information; using the image fusion feature representation as a second denoising condition, the second noise image is subjected to denoising processing through the image transformation diffusion model to obtain the model surface rendering information corresponding to the at least two projected geometry images respectively.

[0370] In an optional embodiment, the rendering module 1040 is further configured to obtain vertex coordinates corresponding to at least one model vertex in the first three-dimensional model, wherein the vertex coordinates are the position information of the model vertex in the three-dimensional space where the first three-dimensional model is located; obtain the camera optical center coordinates corresponding to the virtual camera, wherein the virtual camera is used to observe the three-dimensional space; obtain the viewing direction corresponding to at least one model vertex based on the camera optical center coordinates and the vertex coordinates, wherein the viewing direction is used to characterize the direction from which the model vertex is observed from the virtual camera; and perform texture rendering on the model surface of the first three-dimensional model with reference to the viewing direction and through the model surface rendering information corresponding to the at least two projected geometry maps to obtain the second three-dimensional model.

[0371] In an optional embodiment, the rendering module 1040 is further configured to determine the texture coordinates corresponding to at least two model surface points in the first three-dimensional model based on the model unfolding state of the first three-dimensional model, wherein the model surface points include the model vertices; determine the rendering sub-information corresponding to the texture coordinates of the at least two model surface points from the model surface rendering information corresponding to the at least two projected geometry diagrams, wherein the nth rendering sub-information is used to render the model surface point corresponding to the nth texture coordinate, where n is a positive integer; and, with the viewing direction as a reference, render the model surface point corresponding to the texture coordinate based on the correspondence between the texture coordinate and the rendering sub-information, thereby obtaining the second three-dimensional model.

[0372] In an optional embodiment, the rendering module 1040 is further configured to obtain the first texture coordinates corresponding to the first model surface points; when at least two first rendering sub-information corresponding to the first texture coordinates are determined from the at least two model surface rendering information, the rendering weight corresponding to the viewing direction is obtained, and the rendering weight is used to balance the rendering contribution of the at least two first rendering sub-information to the first model surface points; and the rendering sub-information corresponding to the first texture coordinates is obtained by adjusting the at least two first rendering sub-information through the rendering weight.

[0373] In an optional embodiment, the acquisition module 1010 is further configured to acquire the first 3D model and rendering text, wherein the rendering text is used to describe the adjustment of the display state of the first 3D model in text form, and the rendering guidance information includes the rendering text; or, acquire the first 3D model and the rendering text; acquire a third 3D model that conforms to the rendering text, wherein the rendering guidance information includes the third 3D model; or, acquire the first 3D model and a fourth 3D model, wherein the fourth 3D model is a 3D model displayed in a first display state, the first display state being the display state after guiding the adjustment of the first 3D model, and the rendering guidance information including the fourth 3D model.

[0374] In an optional embodiment, the acquisition module 1010 is further configured to acquire a fifth three-dimensional model representing the first display state, the fifth three-dimensional model including at least one of the third three-dimensional model and the fourth three-dimensional model; perform a delighting process on the fifth three-dimensional model to obtain a sixth three-dimensional model corresponding to the fifth three-dimensional model as the rendering guidance information, the sixth three-dimensional model being used to characterize the surface material properties when the first display state is displayed after removing the influence of other lighting besides ambient lighting.

[0375] In an optional embodiment, the acquisition module 1010 is further configured to acquire the lighting distribution information corresponding to the fifth three-dimensional model, the lighting distribution information including at least one of ambient lighting information, direct lighting information and indirect lighting information; and perform delighting processing on the fifth three-dimensional model based on the lighting distribution information to obtain the sixth three-dimensional model as the rendering guidance information.

[0376] In an optional embodiment, the projection module 1020 is further configured to, during the process of projecting the first three-dimensional model onto the view spaces corresponding to the at least two projection viewpoints respectively, acquire pixel information corresponding to at least two model pixels in the first three-dimensional model, the pixel information including at least one of pixel position, pixel normal and pixel depth; and render the projection geometry corresponding to the at least two projection viewpoints respectively using the pixel information, the projection geometry including at least one of three-dimensional coordinate map, normal map and depth map.

[0377] In summary, during the process of guiding the display state of the first 3D model through rendering guidance information, corresponding projected geometry is obtained from at least two projection viewpoints. This, combined with the rendering guidance information and the at least two projected geometry, generates model surface rendering information corresponding to each of the at least two projection viewpoints. Therefore, the generation process of model surface rendering information fully considers the correlation between at least two viewpoints, ensuring that the changes in the display state of the rendered second 3D model observed from different angles are consistent. Furthermore, by rendering the model surface rendering information corresponding to at least two projection viewpoints, the display state of the model surface when observing the first 3D model from the projection viewpoint is improved, enhancing the rendering accuracy and display effect consistency of the second 3D model.

[0378] It should be noted that the 3D model rendering device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the 3D model rendering device and the 3D model rendering method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0379] Figure 11 This illustration shows a schematic diagram of the structure of a server provided in an exemplary embodiment of this application. Specifically, it includes the following structure.

[0380] Server 1100 includes a Central Processing Unit (CPU) 1101, a system memory 1104 including Random Access Memory (RAM) 1102 and Read Only Memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. Server 1100 also includes a mass storage device 1106 for storing the operating system 1113, application programs 1114, and other program modules 1115.

[0381] Mass storage device 1106 is connected to central processing unit 1101 via a mass storage controller (not shown) connected to system bus 1105. Mass storage device 1106 and its associated computer-readable media provide non-volatile storage for server 1100. That is, mass storage device 1106 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0382] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1104 and mass storage device 1106 described above can be collectively referred to as memory.

[0383] According to various embodiments of this application, server 1100 can also be connected to a remote computer on a network, such as the Internet. That is, server 1100 can be connected to network 1112 via network interface unit 1111 connected to system bus 1105, or it can also use network interface unit 1111 to connect to other types of networks or remote computer systems (not shown).

[0384] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0385] Embodiments of this application also provide a computer device including a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the rendering method for the three-dimensional model provided in the above-described method embodiments. Optionally, the computer device may be a terminal or a server.

[0386] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the rendering method of the three-dimensional model provided in the above-described method embodiments.

[0387] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the three-dimensional model rendering methods described in the above embodiments.

[0388] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0389] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0390] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for rendering a three-dimensional model, characterized in that, The method includes: Obtain the first 3D model and the rendering guidance information for the first 3D model; The first three-dimensional model is projected to obtain at least two projected geometric figures, which correspond to different projection viewpoints and respectively represent different geometric shapes. The rendering guidance information and the at least two projected geometry maps are respectively fused to generate model surface rendering information corresponding to the at least two projected geometry maps. The model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projection view. The model surface of the first three-dimensional model is rendered using the model surface rendering information corresponding to the at least two projected geometry diagrams to obtain the second three-dimensional model.

2. The method according to claim 1, characterized in that, The step of fusing the rendering guidance information and the at least two projected geometry maps to generate model surface rendering information corresponding to the at least two projected geometry maps includes: The rendering guidance information and the i-th projection geometry of the at least two projection geometry are fused to generate the model surface color image corresponding to the i-th projection viewpoint; the rendering guidance information corresponds to the first lighting condition, and the model surface color image corresponding to the i-th projection viewpoint is used to describe the surface color of the model surface observed from the i-th projection viewpoint under the first lighting condition, where i is a positive integer; Based on the model surface color images corresponding to the at least two projection viewpoints, the model surface rendering information corresponding to the at least two projection geometry maps is generated.

3. The method according to claim 2, characterized in that, The step of fusing the rendering guidance information and the i-th projection geometry of the at least two projection geometry maps to generate the model surface color image corresponding to the i-th projection viewpoint includes: Obtain the information feature representation corresponding to the rendering guidance information; Obtain the geometric feature representation corresponding to the i-th projected geometry; By fusing the information feature representation and the geometric feature representation corresponding to the i-th projected geometry, the i-th first feature representation is obtained; The model surface color image corresponding to the i-th projection viewpoint is generated based on the i-th first feature representation.

4. The method according to claim 3, characterized in that, The fusion of the information feature representation and the geometric feature representation corresponding to the i-th projected geometry to obtain the i-th first feature representation includes: Obtain the geometric feature representations corresponding to the at least two projected geometric figures respectively; Perform feature concatenation on the at least two geometric feature representations to obtain a geometric concatenated feature representation; Perform cross-attention analysis on the information feature representation and the geometric splicing feature representation to obtain an attention splicing feature representation, wherein the attention splicing feature representation includes attention sub-feature representations corresponding to the at least two geometric feature representations respectively; Perform self-attention analysis on the attention sub-feature representations corresponding to the at least two geometric feature representations respectively to obtain self-attention feature representations corresponding to the at least two attention sub-feature representations respectively. The self-attention feature representations corresponding to the at least two attention sub-feature representations respectively include the self-attention feature representation corresponding to the i-th projection viewpoint. The self-attention feature representation corresponding to the i-th projection viewpoint is used as the i-th first feature representation corresponding to the i-th projection viewpoint.

5. The method according to claim 3, characterized in that, The fusion of the information feature representation and the geometric feature representation corresponding to the i-th projected geometry to obtain the i-th first feature representation includes: Perform cross-attention analysis on the geometric feature representation and the information feature representation corresponding to the i-th projected geometry to obtain the first attention feature representation corresponding to the i-th geometric feature representation; Obtain the geometric feature representations corresponding to the at least two projected geometry maps, including the geometric feature representation corresponding to the i-th projected geometry map; Self-attention analysis is performed on the first attention feature representations corresponding to at least two geometric feature representations to obtain second attention feature representations corresponding to at least two first attention feature representations. The second attention feature representation is an enhanced feature representation based on the at least two first attention feature representations. The second attention feature representation corresponding to the i-th projection viewpoint is used as the i-th first feature representation corresponding to the i-th projection viewpoint.

6. The method according to claim 3, characterized in that, The step of generating the model surface color image corresponding to the i-th projection viewpoint based on the i-th first feature representation includes: A multi-view diffusion model and a first noise image are obtained, wherein the multi-view diffusion model is a pre-trained model for obtaining color images corresponding to the projection viewpoints; The first feature is represented as the first denoising condition corresponding to the first projection viewpoint. The first noise image is denoised by the multi-view diffusion model to obtain the model surface color image corresponding to the first projection viewpoint.

7. The method according to claim 2, characterized in that, The step of generating model surface rendering information corresponding to the at least two projected geometry maps based on the model surface color images corresponding to the at least two projection viewpoints includes: Obtain the image feature representations corresponding to at least two model surface color images; By fusing the image feature representations corresponding to the at least two model surface color images respectively, an image fusion feature representation is obtained; The model surface rendering information corresponding to the at least two projected geometry maps is obtained based on the image fusion feature representation. The model surface rendering information includes at least one of albedo information, metallicity information, and roughness information.

8. The method according to claim 7, characterized in that, The step of obtaining the model surface rendering information corresponding to the at least two projected geometry maps based on the image fusion feature representation includes: Obtain an image transformation diffusion model and a second noise image, wherein the image transformation diffusion model is a pre-trained model used to generate model rendering information; Using the image fusion feature as the second denoising condition, the second noisy image is denoised using the image conversion diffusion model to obtain the model surface rendering information corresponding to the at least two projected geometry maps.

9. The method according to any one of claims 1 to 8, characterized in that, The step of rendering the model surface of the first three-dimensional model using the model surface rendering information corresponding to the at least two projected geometry maps to obtain the second three-dimensional model includes: Obtain the vertex coordinates corresponding to at least one model vertex in the first three-dimensional model, wherein the vertex coordinates are the position information of the model vertex in the three-dimensional space of the first three-dimensional model; Obtain the optical center coordinates of the virtual camera, which is used to observe the three-dimensional space. Based on the camera optical center coordinates and the vertex coordinates, the viewing direction corresponding to each of the at least one model vertex is obtained, and the viewing direction is used to characterize the direction from which the model vertex is observed from the virtual camera; Using the viewing direction as a reference, texture rendering is performed on the surface of the first three-dimensional model using the model surface rendering information corresponding to the at least two projected geometry diagrams, to obtain the second three-dimensional model.

10. The method according to claim 9, characterized in that, The step of performing texture rendering on the surface of the first 3D model using the model surface rendering information corresponding to the at least two projected geometry maps, with reference to the viewing direction, to obtain the second 3D model includes: Based on the unfolded state of the first three-dimensional model, determine the texture coordinates corresponding to at least two model surface points in the first three-dimensional model, wherein the model surface points include the model vertices; From the model surface rendering information corresponding to the at least two projected geometry maps respectively, determine the rendering sub-information of the texture coordinates corresponding to the at least two model surface points respectively, and the nth rendering sub-information is used to render the model surface point corresponding to the nth texture coordinate, where n is a positive integer; Using the viewing direction as a reference, and based on the correspondence between the texture coordinates and the rendering sub-information, the model surface points corresponding to the texture coordinates are rendered using the rendering sub-information to obtain the second three-dimensional model.

11. The method according to claim 10, characterized in that, Before obtaining the second 3D model by rendering the model surface points corresponding to the texture coordinates using the rendering sub-information based on the correspondence between the texture coordinates and the rendering direction, the method further includes: Obtain the first texture coordinates corresponding to the surface points of the first model; When at least two first rendering sub-informations corresponding to the first texture coordinates are determined from the at least two model surface rendering information, the rendering weight corresponding to the viewing direction is obtained, and the rendering weight is used to balance the rendering contribution of the at least two first rendering sub-informations to the first model surface points. By adjusting at least two first rendering sub-information values ​​using the rendering weights, rendering sub-information corresponding to the first texture coordinates is obtained.

12. The method according to any one of claims 1 to 8, characterized in that, The acquisition of the first 3D model and the rendering guidance information for the first 3D model includes: Obtain the first 3D model and rendered text, wherein the rendered text describes in text form the adjustments made to the display state of the first 3D model, and the rendering guidance information includes the rendered text; or... Obtain the first 3D model and the rendered text; obtain a third 3D model that matches the rendered text, wherein the rendering guidance information includes the third 3D model; or, Obtain the first three-dimensional model and the fourth three-dimensional model. The fourth three-dimensional model is a three-dimensional model displayed in a first display state. The first display state is the display state after guiding the adjustment of the first three-dimensional model. The rendering guidance information includes the fourth three-dimensional model.

13. The method according to claim 12, characterized in that, The method further includes: Obtain a fifth three-dimensional model representing the first display state, wherein the fifth three-dimensional model includes at least one of the third three-dimensional model and the fourth three-dimensional model; The fifth 3D model is subjected to a delighting process to obtain a sixth 3D model corresponding to the fifth 3D model, which is used as the rendering guidance information. The sixth 3D model is used to characterize the surface material properties when the first display state is displayed after removing the influence of other lighting besides ambient lighting.

14. The method according to claim 13, characterized in that, The step of performing delighting processing on the fifth 3D model to obtain a sixth 3D model corresponding to the fifth 3D model as the rendering guidance information includes: Obtain the illumination distribution information corresponding to the fifth three-dimensional model, wherein the illumination distribution information includes at least one of ambient illumination information, direct illumination information, and indirect illumination information; Based on the lighting distribution information, the fifth 3D model is subjected to delighting processing to obtain the sixth 3D model as the rendering guidance information.

15. The method according to any one of claims 1 to 8, characterized in that, The projection of the first three-dimensional model to obtain at least two projected geometric figures includes: During the process of projecting the first three-dimensional model onto the view space corresponding to the at least two projection perspectives, pixel information corresponding to at least two model pixels in the first three-dimensional model is obtained. The pixel information includes at least one of pixel position, pixel normal, and pixel depth. The pixel information is used to render the projection geometry corresponding to the at least two projection viewpoints, and the projection geometry includes at least one of a three-dimensional coordinate map, a normal map, and a depth map.

16. A rendering device for a three-dimensional model, characterized in that, The device includes: The acquisition module is used to acquire the first 3D model and the rendering guidance information for the first 3D model; The projection module is used to project the first three-dimensional model to obtain at least two projected geometric figures, wherein the at least two projected geometric figures correspond to different projection perspectives and respectively refer to different geometric shapes; The fusion module is used to fuse the rendering guidance information and the at least two projected geometry maps respectively to generate model surface rendering information corresponding to the at least two projected geometry maps respectively. The model surface rendering information is used to render the display state of the model surface of the first three-dimensional model under the projection view. The rendering module is used to render the model surface of the first three-dimensional model using the model surface rendering information corresponding to the at least two projected geometry diagrams, to obtain the second three-dimensional model.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the rendering method of the three-dimensional model as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the rendering method of the three-dimensional model as described in any one of claims 1 to 15.

19. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the rendering method of the three-dimensional model as described in any one of claims 1 to 15.

Images