Construction method of 3D reconstruction large model, controller, system, medium and product

Abstract

The invention provides a construction method of a 3D reconstruction large model, a controller, a system, a medium and a product, and the method comprises the steps: obtaining a training image, and carrying out the rendering of the training image based on grid rasterization, and obtaining a photometric stereo image; inputting the photometric stereo image into a preset display network to obtain a three-dimensional representation of the photometric stereo image; performing picture rendering on the three-dimensional representation to obtain various rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map; carrying out loss calculation according to various rendering images, albedo images and photometric stereo images through a preset loss function to obtain a loss value; and optimizing the display network according to the loss value to obtain a constructed 3D reconstruction large model. Therefore, the loss calculation is carried out through various rendering images, albedo images and photometric stereo images, so that the explicit network is optimized according to the loss value to obtain the constructed 3D reconstruction large model, and the precision and quality of the construction of the 3D reconstruction large model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional modeling, and in particular to a method, controller, system, medium and product for constructing a large 3D reconstruction model. Background Art

[0002] In the related art, the methods for constructing image-generated 3D models are mainly divided into three categories. The first method relies on the prior knowledge of the 2D diffusion model. It generates images of multiple views and reconstructs them based on these multiple views to achieve the conversion from images to 3D models. This method makes full use of the advantages of the 2D diffusion model in image generation, but the generated multiple views may have consistency problems, which may cause the final reconstructed 3D model to have an uneven surface or other defects.

[0003] The second type of method is based on the technology of generating large 3D models. It relies on the training of geometric variational autoregressive models and latent space diffusion models to achieve high-quality 3D model generation. However, its effect depends largely on the number of available 3D models and the accuracy of the labels. In addition, the second stage of this method consumes a lot of resources, and the generated 3D model may not be completely consistent with the given image input. It also faces optimization challenges.

[0004] The third method is to use a large forward 3D reconstruction model, which uses a pre-trained model to project the image into 3D space and optimizes the entire framework through the rendering loss of the 2D image to achieve a 3D reconstruction model that is highly consistent with the input image and only requires 2D images as supervision information. However, this method often performs poorly in terms of local details and is sensitive to the complex appearance of the input image, which affects the accuracy and authenticity of the reconstructed model.

[0005] Therefore, how to realize the construction of large 3D reconstruction models is the main problem currently facing the industry. Summary of the invention

[0006] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a method, controller, system, medium and product for constructing a 3D reconstruction large model, aiming to improve the accuracy and quality of the 3D reconstruction large model.

[0007] In a first aspect, an embodiment of the present application provides a method for constructing a 3D reconstructed large model, the method comprising:

[0008] Acquire a training image, and render the training image based on grid rasterization to obtain a photometric stereo image;

[0009] Inputting the photometric stereo image into a preset explicit network to obtain a three-dimensional representation of the photometric stereo image;

[0010] Performing image rendering on the three-dimensional representation to obtain a plurality of rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map;

[0011] Performing loss calculation according to the various rendered images, the albedo map, and the photometric stereo image using a preset loss function to obtain a loss value;

[0012] The explicit network is optimized according to the loss value to obtain a constructed 3D reconstructed large model.

[0013] According to some embodiments of the present application, rendering the training image based on grid rasterization to obtain a photometric stereo image includes:

[0014] Collecting multiple light maps, and obtaining lighting information according to the light maps;

[0015] Acquire material properties of a preset network model, and rasterize the network model to obtain rasterization information of the network model, wherein the material properties include albedo, metallicity, and roughness;

[0016] Calculate the diffuse reflection albedo and the specular reflection albedo according to the albedo and the metalness;

[0017] Defining a sampling range of the metalness and the roughness;

[0018] The training image is subjected to diffuse reflection and specular reflection according to the illumination information, the rasterization information, the diffuse reflection albedo, the specular reflection albedo, and the sampling range to obtain the photometric stereo images of multiple viewing angles.

[0019] According to some embodiments of the present application, the explicit network includes an image encoder, and the step of inputting the photometric stereo image into a preset explicit network to obtain a three-dimensional representation of the photometric stereo image features includes:

[0020] Inputting the photometric stereo image into an image encoder in a preset explicit network for encoding to obtain a first picture feature;

[0021] A three-dimensional representation of the photometric stereo image feature is obtained according to the first picture feature.

[0022] According to some embodiments of the present application, the explicit network further includes a three-plane decoder, and obtaining the three-dimensional representation of the photometric stereo image feature according to the first picture feature includes:

[0023] The first picture feature is input into a three-plane decoder in the explicit network for decoding to obtain a three-dimensional representation of the first picture feature.

[0024] According to some embodiments of the present application, the multiple rendered images include a specular light map, a diffuse light map, a depth map, and a normal map.

[0025] According to some embodiments of the present application, the loss function is:

[0026]

[0027] in, is the loss function, is the mean square error loss function, C is the true photometric stereo image, is the photometric stereo image, λ LPIPS is the first coefficient, To learn the perceptual image patch similarity loss function, a is the real albedo map, is the albedo map, L * is the real specular light map and the real diffuse light map, are the specular light map and the diffuse light map, n is the real normal map, is the normal graph, λ normal is the second coefficient, m is the real mask, is the predicted mask, λ reg is the third coefficient, is the geometric constraint loss function, λ depth is the fourth coefficient, d is the real depth map, is the depth map, λ mask is the fifth coefficient.

[0028] In a second aspect, an embodiment of the present application provides a controller comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes a method for constructing a 3D reconstructed large model according to the first aspect above when running the computer program.

[0029] In a third aspect, an embodiment of the present application provides a system for constructing a 3D reconstructed large model, including the controller of the second aspect described above.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute a method for constructing a 3D reconstructed large model as described in the first aspect above.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program or computer instructions, characterized in that the computer program or the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the computer device executes a method for constructing a 3D reconstructed large model as described in the first aspect above.

[0032] According to the technical solution of the embodiment of the present application, at least the following beneficial effects are achieved: the embodiment of the present application proposes a method, controller, system, medium and product for constructing a 3D reconstruction large model, the method comprising: obtaining a training image, rendering the training image based on grid rasterization to obtain a photometric stereo image; inputting the photometric stereo image into a preset display network to obtain a three-dimensional representation of the photometric stereo image; rendering the three-dimensional representation to obtain a plurality of rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map; calculating the loss according to various rendered images, albedo maps and photometric stereo images through a preset loss function to obtain a loss value; optimizing the display network according to the loss value to obtain a constructed 3D reconstruction large model. Therefore, the present application calculates the loss through various rendered images, albedo maps and photometric stereo images, thereby optimizing the explicit network according to the loss value to obtain a constructed 3D reconstruction large model, thereby improving the accuracy and quality of the construction of the 3D reconstruction large model.

[0033] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0035] Figure 1 It is a flow chart of a method for constructing a 3D reconstructed large model provided by an embodiment of the present application;

[0036] Figure 2 is a flow chart of a method for constructing a 3D reconstructed large model provided in another embodiment of the present application;

[0037] Figure 3 is a flow chart of a method for constructing a 3D reconstructed large model provided in another embodiment of the present application;

[0038] Figure 4This is an overall flow chart of a method for constructing a 3D reconstructed large model provided by an embodiment of the present application;

[0039] Figure 5 It is an overall flow chart of a method for constructing a 3D reconstructed large model provided in another embodiment of the present application;

[0040] Figure 6 is a schematic diagram of a 3D object provided by an embodiment of the present application;

[0041] Figure 7 is a schematic diagram of a 3D object provided by another embodiment of the present application;

[0042] Figure 8 is a schematic diagram of a 3D object provided by another embodiment of the present application;

[0043] Fig. 9 It is a schematic diagram of a controller for executing a method for constructing a 3D reconstructed large model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0045] In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0046] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0047] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0048] In some cases, the methods for constructing image-generated 3D models can be mainly divided into three categories. The first method relies on the prior knowledge of the 2D diffusion model. It generates images of multiple views and reconstructs them based on these multiple views to achieve the conversion from images to 3D models. This method makes full use of the advantages of the 2D diffusion model in image generation, but the generated multiple views may have consistency problems, which may cause the final reconstructed 3D model to have an uneven surface or other defects.

[0049] The second type of method is based on the technology of generating large 3D models. It relies on the training of geometric variational autoregressive models and latent space diffusion models to achieve high-quality 3D model generation. However, its effect depends largely on the number of available 3D models and the accuracy of the labels. In addition, the second stage of this method consumes a lot of resources, and the generated 3D model may not be completely consistent with the given image input. It also faces optimization challenges.

[0050] The third method is to use a large forward 3D reconstruction model, which uses a pre-trained model to project the image into 3D space and optimizes the entire framework through the rendering loss of the 2D image to achieve a 3D reconstruction model that is highly consistent with the input image and only requires 2D images as supervision information. However, this method often performs poorly in terms of local details and is sensitive to the complex appearance of the input image, which affects the accuracy and authenticity of the reconstructed model.

[0051] Therefore, how to realize the construction of large 3D reconstruction models is the main problem currently facing the industry.

[0052] Based on the above situation, the embodiments of the present application propose a method, controller, system, medium and product for constructing a 3D reconstructed large model, aiming to improve the accuracy and quality of constructing a 3D reconstructed large model.

[0053] The following is a further description of various embodiments of a method for constructing a 3D reconstructed large model of the present application in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, Figure 1 It is a flowchart of a method for constructing a 3D reconstructed large model provided by an embodiment of the present application; the method for constructing a 3D reconstructed large model may include but is not limited to step S110, step S120, step S130, step S140 and step S150.

[0055] Step S110, obtaining a training image, rendering the training image based on grid rasterization, and obtaining a photometric stereo image;

[0056] Step S120, inputting the photometric stereo image into a preset explicit network to obtain a three-dimensional representation of the photometric stereo image;

[0057] Step S130, performing image rendering on the three-dimensional representation to obtain a plurality of rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map;

[0058] Step S140, performing loss calculation according to various rendered images, albedo maps, and photometric stereo images using a preset loss function to obtain a loss value;

[0059] Step S150: Optimize the explicit network according to the loss value to obtain a constructed 3D reconstructed large model.

[0060] In one embodiment, first, a training image is obtained, and the training image is rendered based on grid rasterization to obtain a photometric stereo image; secondly, the photometric stereo image is input into a preset display network for encoding and decoding to obtain a three-dimensional representation of the photometric stereo image; thirdly, the three-dimensional representation is image rendered to obtain a plurality of rendered images, and the albedo is predicted for the three-dimensional representation to obtain an albedo map; then, a loss calculation is performed on various rendered images, albedo maps, and photometric stereo images through a preset loss function to obtain a loss value; finally, the explicit network is optimized according to the loss value to obtain a constructed 3D reconstruction model. Therefore, the present application performs loss calculations through various rendered images, albedo maps, and photometric stereo images, thereby optimizing the explicit network according to the loss value to obtain a constructed 3D reconstruction model, thereby improving the accuracy and quality of the construction of the 3D reconstruction model.

[0061] It can be understood that the present application performs online rendering through grid rasterization and physically-based real-time rendering technology. During the rendering process, the material, environment map and camera pose will change randomly, thereby rendering a multi-view photometric stereo image.

[0062] It can be understood that the display network of the present application supports rasterization rendering, which is faster and occupies less video memory than traditional voxel rendering; secondly, the display network not only supports the rendering of photometric information, but also can render geometric information, such as depth maps and normal maps. These geometric information provides additional optimization constraints; in addition, the display network supports physically based rendering, which can more accurately simulate reflected light, thereby better reconstructing highly reflective objects.

[0063] It can be understood that grid rasterization can quickly obtain surface points, masks, normal vectors and depth information. Combining this information with the real-time rendering method of segmentation and summation, it can render color, diffuse light map, specular light map and other photometric information.

[0064] It can be understood that the various rendered images include a specular light map, a diffuse light map, a depth map, and a normal map.

[0065] It can be understood that the loss function is:

[0066]

[0067] in, is the loss function, is the mean square error loss function, C is the true photometric stereo image, is the photometric stereo image, λ LPIPS is the first coefficient, To learn the perceptual image patch similarity loss function, a is the real albedo map, is the albedo map, L * is the real specular light map and the real diffuse light map, are the specular light map and the diffuse light map, n is the real normal map, is the normal graph, λ normal is the second coefficient, m is the real mask, is the predicted mask, λ reg is the third coefficient, is the geometric constraint loss function, λ depth is the fourth coefficient, d is the real depth map, is the depth map, λ mask is the fifth coefficient.

[0068] It can be understood that the multi-objective optimization loss function greatly improves the model's ability to restore geometry, thereby improving the quality of building large 3D reconstructed models.

[0069] In addition, if Figure 2 As shown, Figure 2 It is a flowchart of a method for constructing a 3D reconstructed large model provided by another embodiment of the present application; regarding the rendering of the training image based on grid rasterization in the above-mentioned step S110 to obtain a photometric stereo image, it may include but is not limited to step S210, step S220, step S230, step S240 and step S250.

[0070] Step S210, collecting multiple light maps, and obtaining lighting information according to the light maps;

[0071] Step S220, obtaining material properties of a preset network model, and rasterizing the network model to obtain rasterization information of the network model, where the material properties include albedo, metallicity, and roughness;

[0072] Step S230, calculating the diffuse reflection albedo and the specular reflection albedo according to the albedo and the metalness;

[0073] Step S240, defining the sampling range of metalness and roughness;

[0074] Step S250: Perform diffuse reflection and specular reflection on the training image according to the illumination information, rasterization information, diffuse reflection albedo, specular reflection albedo and sampling range to obtain photometric stereo images of multiple viewing angles.

[0075] In one embodiment, first, multiple light maps are collected to obtain lighting information based on the light maps; then, the material properties of a preset network model are obtained, and the network model is rasterized. After the network model is rasterized, a rasterization message of the network model is obtained, wherein the material properties include albedo, metallicity, and roughness; then, the diffuse albedo and the specular albedo are calculated based on the albedo and metallicity; in addition, the sampling range of the metallicity and the roughness is defined; finally, the training image is diffusely reflected and specularly reflected based on the obtained lighting information, rasterization information, diffuse albedo, specular albedo, and sampling range, thereby obtaining photometric stereo images of multiple perspectives.

[0076] It can be understood that the rasterization information of the network model obtained after rasterizing the network model is indispensable key information in the subsequent rendering process, and together constitutes the basis of the rendering pipeline, so that realistic three-dimensional graphics can be generated.

[0077] It can be understood that the diffuse albedo is given by the formula a d =(1-m)a, where a d is the diffuse albedo, m is the metalness, and a is the albedo.

[0078] It can be understood that the specular albedo is given by the formula a s =((1-m)*0.04+m*a, where a S is the specular albedo.

[0079] It can be understood that the physical-based real-time rendering of the present application includes two parts: diffuse reflection and specular reflection. The diffuse reflection is calculated by formula C d (x,ω0)=a d (a,m)L diff (x,n,M d ) is obtained, and the specular reflection is obtained by formula C s (x,ω0)=a s (a,m,n,ω0)L spec (x,h,ω0,ρ,M s ) is obtained, so that after diffuse reflection and specular reflection of the training image, photometric stereo images of various views are obtained, where C d is the diffuse color, x is the surface point, ω0 is the sight direction, L diff is diffuse light, n is the normal vector, Md is diffuse ambient light, C s is the specular color, L spec is the specular reflected light, ρ is the roughness, M s Specular reflection of ambient light.

[0080] It can be understood that the embodiment of the present application randomly selects metalness, roughness and lighting information from predefined options to perform diffuse reflection and specular reflection on the training image.

[0081] It is understandable that the range of metallicity and roughness can be 0-1, increasing by 0.1 each time, with a total of 121 possible combinations. The embodiments of the present application do not specifically limit the range of metallicity and roughness.

[0082] It can be understood that regarding the above-mentioned collection of multiple light maps, it can be 600 light maps, it can be 650 light maps, and it can be collected according to actual needs. The embodiment of the present application does not specifically limit the number of light maps collected.

[0083] In addition, if Figure 3 As shown, Figure 3 It is a flowchart of a method for constructing a 3D reconstructed large model provided by another embodiment of the present application; regarding the above-mentioned step S120, it may include but is not limited to step S310 and step S320.

[0084] Step S310: inputting the photometric stereo image into an image encoder in a preset explicit network for encoding to obtain a first image feature;

[0085] Step S320: Obtain a three-dimensional representation of the photometric stereo image feature according to the first image feature.

[0086] In one embodiment, the embodiment of the present application inputs the rendered photometric stereo image into an image encoder in a preset display network for encoding, thereby obtaining a first image feature, and further obtaining a three-dimensional representation of the photometric stereo image feature.

[0087] In addition, the above step S320 specifically includes the following steps:

[0088] The first image feature is input into the three-plane decoder in the explicit network for decoding to obtain a three-dimensional representation of the first image feature.

[0089] It will be appreciated that the display network includes an image encoder and a three-plane decoder.

[0090] Based on the above-mentioned methods for constructing a 3D reconstructed large model in each of the embodiments, overall embodiments of the method for constructing a 3D reconstructed large model of the present application are proposed below.

[0091] like Figure 4 and 5 As shown, Figure 4 This is an overall flow chart of a method for constructing a 3D reconstruction large model provided by an embodiment of the present application. Figure 5 This is an overall flow chart of a method for constructing a 3D reconstructed large model provided in another embodiment of the present application.

[0092] Step S401, obtaining a training image;

[0093] Step S402: Collect multiple light maps, and obtain light information according to the light maps;

[0094] Step S403, obtaining the material properties of the preset network model, and rasterizing the network model to obtain rasterization information of the network model, where the material properties include albedo, metallicity, and roughness;

[0095] Step S404, calculating the diffuse reflection albedo and the specular reflection albedo according to the albedo and the metalness;

[0096] Step S405, defining the sampling range of metalness and roughness;

[0097] Step S406, performing diffuse reflection and specular reflection on the training image according to the illumination information, rasterization information, diffuse reflection albedo, specular reflection albedo and sampling range to obtain photometric stereo images of multiple viewing angles;

[0098] Step S407: input the photometric stereo image into an image encoder in a preset explicit network for encoding to obtain a first image feature;

[0099] Step S408: input the first picture feature into the three-plane decoder in the explicit network for decoding to obtain a three-dimensional representation of the first picture feature;

[0100] Step S409: performing image rendering on the three-dimensional representation to obtain a plurality of rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map;

[0101] Step S410, performing loss calculation according to various rendered images, albedo maps, and photometric stereo images using a preset loss function to obtain a loss value;

[0102] Step S411: optimize the explicit network according to the loss value to obtain a constructed 3D reconstructed large model.

[0103] It is worth noting that the present application calculates losses through various rendered images, albedo maps, and photometric stereo images, and then optimizes the explicit network according to the loss value to obtain a constructed 3D reconstructed large model, thereby improving the accuracy and quality of the 3D reconstructed large model.

[0104] It can be understood that since the embodiments of the present application utilize multi-view photometric stereo images as the input and supervision of the model, the model's robustness to the appearance of the input image and its ability to restore high-quality local details are significantly improved.

[0105] In order to better understand the solution of the embodiment of the present application, the method for constructing a 3D reconstructed large model provided by the embodiment of the present application is explained below through several specific application examples.

[0106] Example 1

[0107] like Figure 6 As shown, Figure 6 It is a schematic diagram of a 3D object provided by an embodiment of the present application.

[0108] In one embodiment, downstream applications such as relighting and material editing may be performed on the reconstructed 3D object.

[0109] It can be understood that the 3D reconstructed large model of the embodiment of the present application can be rendered under different environment maps so that the object presents different appearance effects.

[0110] Example 2

[0111] like Figure 7 As shown, Figure 7 It is a schematic diagram of a 3D object provided by another embodiment of the present application.

[0112] In one embodiment, the 3D reconstructed large model of the embodiment of the present application can change the material of the object.

[0113] Example 3

[0114] like Figure 8 As shown, Figure 8 It is a schematic diagram of a 3D object provided by another embodiment of the present application.

[0115] In one embodiment, the 3D reconstructed large model of the embodiment of the present application can gradually increase the metallicness of the object so that the object appears metallic.

[0116] Based on the above-mentioned methods for constructing a large 3D reconstruction model, various embodiments of the controller, a system for constructing a large 3D reconstruction model, a computer-readable storage medium and a computer program product of the present application are respectively proposed below.

[0117] like Fig. 9 As shown, Fig. 9700 is a schematic diagram of a controller for executing a method for constructing a 3D reconstructed large model provided by an embodiment of the present application. The controller 700 implemented in the present application includes: a processor 710, a memory 720, and a computer program stored in the memory 720 and executable on the processor 710, wherein: Fig. 9 In the figure, a processor 710 and a memory 720 are taken as an example.

[0118] The processor 710 and the memory 720 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0119] The memory 720, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 720 optionally includes a memory 720 remotely arranged relative to the processor 710, and these remote memories 720 can be connected to the controller 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] Those skilled in the art will understand that Fig. 9 The device structure shown in the figure does not constitute a limitation on the controller 700, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0121] exist Fig. 9 In the controller 700 shown, the processor 710 can be used to call the control program stored in the memory 720, so as to implement the above-mentioned method for constructing a 3D reconstructed large model. Specifically, the non-transient software program and instructions required to implement the above-mentioned method for constructing a 3D reconstructed large model are stored in the memory 720, and when executed by the processor 710, the above-mentioned method for constructing a 3D reconstructed large model is executed.

[0122] It is worth noting that since the controller 700 of the embodiment of the present application can execute a method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments, the specific implementation manner and technical effects of the controller 700 of the embodiment of the present application can refer to the specific implementation manner and technical effects of a method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments.

[0123] In addition, an embodiment of the present application further provides a system for constructing a 3D reconstructed large model, and the system for constructing a 3D reconstructed large model includes the controller of the above embodiment.

[0124] It is worth noting that since the construction system of the 3D reconstruction large model of the embodiment of the present application includes the controller of the above-mentioned embodiment, and the controller of the above-mentioned embodiment can execute a construction method of a 3D reconstruction large model of any of the above-mentioned embodiments, the specific implementation manner and technical effects of the multi-connected system containing domestic hot water of the embodiment of the present application can refer to the specific implementation manner and technical effects of a construction method of a 3D reconstruction large model of any of the above-mentioned embodiments.

[0125] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned method for constructing a 3D reconstruction model. Figures 1 to 4 The method steps in .

[0126] It is worth noting that since the computer-readable storage medium of the embodiment of the present application can execute a method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments, the specific implementation manner and technical effects of the computer-readable storage medium of the embodiment of the present application can refer to the specific implementation manner and technical effects of the method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments.

[0127] In addition, an embodiment of the present application further provides a computer program product, including a computer program or a computer instruction, the computer program or the computer instruction is stored in a computer-readable storage medium, the processor of the computer device reads the computer program or the computer instruction from the computer-readable storage medium, and the processor executes the computer program or the computer instruction, so that the computer device executes the above-mentioned method for constructing a 3D reconstruction large model. Exemplarily, the above-described Figures 1 to 4 The method steps in .

[0128] It is worth noting that since the computer program product of the embodiment of the present application can execute a method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments, the specific implementation manner and technical effects of the computer program product of the embodiment of the present application can refer to the specific implementation manner and technical effects of the method for constructing a 3D reconstructed large model in any of the above-mentioned embodiments.

[0129] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0130] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0131] In several embodiments provided in the present application, it should be understood that the disclosed systems, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of apparatuses or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0132] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0133] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions under the shared conditions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for constructing a 3D reconstructed large model, characterized in that: The method comprises: Acquire a training image, and render the training image based on grid rasterization to obtain a photometric stereo image; Inputting the photometric stereo image into a preset explicit network to obtain a three-dimensional representation of the photometric stereo image; Performing image rendering on the three-dimensional representation to obtain a plurality of rendered images, and performing albedo prediction on the three-dimensional representation to obtain an albedo map; Performing loss calculation according to the various rendered images, the albedo map, and the photometric stereo image using a preset loss function to obtain a loss value; The explicit network is optimized according to the loss value to obtain a constructed 3D reconstructed large model.

2. The method for constructing a 3D reconstruction large model according to claim 1, characterized in that: The step of rendering the training image based on grid rasterization to obtain a photometric stereo image includes: Collecting multiple light maps, and obtaining lighting information according to the light maps; Acquire material properties of a preset network model, and rasterize the network model to obtain rasterization information of the network model, wherein the material properties include albedo, metallicity, and roughness; Calculate the diffuse reflection albedo and the specular reflection albedo according to the albedo and the metalness; Defining a sampling range of the metalness and the roughness; The training image is subjected to diffuse reflection and specular reflection according to the illumination information, the rasterization information, the diffuse reflection albedo, the specular reflection albedo, and the sampling range to obtain the photometric stereo images of multiple viewing angles.

3. The method for constructing a 3D reconstruction large model according to claim 1, characterized in that: The explicit network includes an image encoder, and the step of inputting the photometric stereo image into a preset explicit network to obtain a three-dimensional representation of the photometric stereo image features includes: Inputting the photometric stereo image into an image encoder in a preset explicit network for encoding to obtain a first picture feature; A three-dimensional representation of the photometric stereo image feature is obtained according to the first picture feature.

4. The method for constructing a 3D reconstruction large model according to claim 3, characterized in that: The explicit network also includes a three-plane decoder, and obtaining a three-dimensional representation of the photometric stereo image feature according to the first picture feature includes: The first picture feature is input into a three-plane decoder in the explicit network for decoding to obtain a three-dimensional representation of the first picture feature.

5. The method for constructing a 3D reconstruction large model according to claim 1, characterized in that: The various rendered images include a specular light map, a diffuse light map, a depth map, and a normal map.

6. The method for constructing a 3D reconstruction large model according to claim 5, characterized in that: The loss function is: in, is the loss function, is the mean square error loss function, C is the true photometric stereo image, is the photometric stereo image, λ LPIPS is the first coefficient, To learn the perceptual image patch similarity loss function, a is the real albedo map, is the albedo map, L * is the real specular light map and the real diffuse light map, are the specular light map and the diffuse light map, n is the real normal map, is the normal graph, λ normal is the second coefficient, m is the real mask, is the predicted mask, λ reg is the third coefficient, is the geometric constraint loss function, λ depth is the fourth coefficient, d is the real depth map, is the depth map, λ mask is the fifth coefficient.

7. A controller, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for constructing a 3D reconstructed large model as described in any one of claims 1 to 6 when executing the computer program.

8. A system for constructing a 3D reconstruction large model, characterized in that: Comprising a controller as claimed in claim 7.

9. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method for constructing a 3D reconstructed large model as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program or computer instructions, characterized in that The computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of a computer device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, so that the computer device executes the method for constructing a 3D reconstructed large model as described in any one of claims 1 to 6.

Images