Inverse rendering method and device of fused semantic segmentation considering shadow

Abstract

The invention discloses an inverse rendering method and device of fused semantic segmentation considering shadow, and relates to the field of computer vision and graphics, and the method comprises the steps: taking a group of multi-view photos of a three-dimensional scene as the geometric shape of an input reconstruction scene, and achieving neural implicit surface reconstruction; taking the semantic segmentation map of the sparse view angle image albedo of the three-dimensional scene as input, obtaining a surface point set with an albedo semantic category of the scene, and projecting the surface point set to an image plane to obtain a semantic segmentation map of each view angle image albedo of the scene; taking a group of multi-view photos of a three-dimensional scene and the obtained semantic segmentation map as input, and carrying out light ray sampling on surface points of the scene; forward rendering is carried out based on incident light on sampling light and materials of surface points, illumination and materials are optimized through network training, and accurate illumination and materials are obtained through decomposition. According to the method and the device, the influence of shadow on material estimation can be effectively weakened, so that relatively accurate illumination and material estimation is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision and graphics, and in particular to an inverse rendering method and device for fusion semantic segmentation taking shadows into consideration. Background Art

[0002] Using two-dimensional images taken from the real world to reconstruct high-fidelity three-dimensional scenes in computers can be applied to virtual reality, augmented reality, and digital twins. However, two-dimensional images correspond to a specific state of the real scene. Considering more flexible applications, such as the ability to modify the scene, including relighting, material editing, etc., it is necessary to obtain the material and lighting properties of the real scene. This work is usually called inverse rendering, which refers to restoring the physical properties of the three-dimensional scene such as geometry, material properties, and lighting environment from two-dimensional images. However, there is a lot of uncertainty in correctly decomposing these properties from the results of the joint synthesis of geometry, material, and lighting. Different combinations of these properties may have the same synthesis results. Due to its complexity and ill-posedness, this work has always been a challenging problem in the field of computer vision and graphics.

[0003] Early inverse rendering work was mainly devoted to intrinsic image decomposition, decomposing images into albedo maps and brightness maps, which can be roughly divided into two categories: methods based on manually set prior knowledge and methods based on deep learning. Methods based on manually set prior knowledge often set thresholds based on empirical knowledge to decompose the albedo and brightness of the image; methods based on deep learning usually require albedo label values ​​for supervision, but the albedo of real scenes is usually difficult to obtain. The current implicit neural rendering method has achieved good results in new view synthesis, 3D reconstruction, etc. Its emergence has also brought new ideas for solving problems to inverse rendering. Current related research work mostly uses multi-layer perceptrons (MultilayerPerceptron, MLP for short) to learn the geometry and material of the scene, and uses physically based rendering equations to replace the volume rendering equations in Neural Radiance Fields (NeRF) to deal with this problem.

[0004] However, in inverse rendering, the presence of shadows will interfere with the estimation of materials, and it is easy to interpret shadows as material properties of the scene itself. Some current related work in implicit neural rendering has not directly proposed a solution to this problem. In addition, some studies use prior knowledge, such as using the semantic classification results obtained by the semantic segmentation network to directly constrain the materials of areas belonging to the same category to be the same. Although this method reduces the impact of shadows on material prediction, the two-dimensional semantic segmentation network used requires a large number of semantic segmentation label values ​​from different perspectives, and has weak generalization capabilities, and cannot obtain more accurate semantic segmentation results for each perspective of the three-dimensional scene. Summary of the invention

[0005] The present application provides an inverse rendering method and device for fused semantic segmentation taking into account shadows, which can effectively reduce the impact of shadows on material estimation, thereby achieving more accurate estimation of lighting and material.

[0006] In a first aspect, an embodiment of the present application provides an inverse rendering method of fused semantic segmentation considering shadows, and the inverse rendering method of fused semantic segmentation considering shadows includes:

[0007] Taking a set of multi-view photos of a 3D scene as input, the geometric shape of the scene, i.e. the signed distance function of the scene, is reconstructed to achieve neural implicit surface reconstruction;

[0008] Taking the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, a surface point set with the albedo semantic category of the scene is obtained, and the surface point set is projected onto the image plane to obtain the semantic segmentation map of the albedo of the scene at each view image;

[0009] Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene;

[0010] Forward rendering is performed based on the incident light on the sampling light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

[0011] In conjunction with the first aspect, in one implementation, geometric reconstruction of a three-dimensional scene specifically includes:

[0012] A pixel on the image forms a ray with the camera, and a finite number of spatial discrete points are sampled along the camera ray to approximate the integral value of the volume rendering integral equation;

[0013] Two multi-layer perceptrons are used to map spatial positions to symbolic distance, geometric feature vectors and colors respectively;

[0014] Perform forward rendering and use the signed distance function as the weight of the volume rendering integral. The signed distance function is optimized by minimizing the reconstruction loss of the rendered color and the observed color.

[0015] In combination with the first aspect, in one implementation, the volume rendering integral equation is specifically expressed as:

[0016]

[0017] Among them, C 1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c iRepresents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

[0018] In combination with the first aspect, in one implementation, the step of taking a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene as input, obtaining a surface point set of the scene with an albedo semantic category, and projecting the surface point set to an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene specifically includes:

[0019] The albedo categories of the sparse view images of the three-dimensional scene are selected for annotation, and the selected sparse view images need to cover all positions of the three-dimensional scene, so as to obtain the semantic segmentation map I of the albedo of the corresponding view image N ;

[0020] Input semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we obtain the point x on the ray where the signed distance function value is 0 i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thus obtaining a surface point set X = {xx = Ω(p), p∈I N}, where x, Ω(p) represent surface points and p represents pixel points;

[0021] Project the surface points in the surface point set onto the image plane to obtain a semantic segmentation map of the albedo of each viewpoint of the scene;

[0022] The obtained semantic segmentation map is denoised using median filtering to obtain the final semantic segmentation map.

[0023] In combination with the first aspect, in one implementation, the surface points in the surface point set are projected onto the image plane, wherein for the projection, the specific process includes:

[0024] Known view image I to be projected j The intrinsic parameter matrix K of the corresponding camera j and the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system;

[0025] For a surface point x i , which is projected onto Ij The pixel point p on j (u j ,v j )=K j ·T j ·x i , thus establishing p i (u i ,v i ) and p j (u j ,v j ), that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

[0026] In combination with the first aspect, in one implementation, forward rendering is performed based on the incident light on the sampling light and the material of the surface point, wherein for the inverse rendering of the three-dimensional scene, the specific process includes:

[0027] Based on the scene's signed distance function, the scene's lighting and materials are modeled and forward rendering is performed;

[0028] Optimize lighting and materials by minimizing reconstruction loss of rendered and observed colors;

[0029] Among them, for forward rendering, forward rendering uses a physically based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as:

[0030]

[0031] in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function is Ω represents a hemisphere centered at the surface point normal vector.

[0032] In combination with the first aspect, in one implementation,

[0033] for The specific calculation method is: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting, wherein the direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene;

[0034] For sampling rays, multiple importance sampling technique is used;

[0035] For ambient light, a spherical Gaussian function is used for modeling, specifically:

[0036]

[0037] Among them, E(w i ) indicates w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, M represents the number of spherical Gaussian functions;

[0038] For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i The outgoing radiance in the direction.

[0039] In combination with the first aspect, in one implementation, for The specific calculation process is:

[0040] For the input of the material network model, the albedo semantic category is encoded using one-hot encoding and combined with surface points and geometric feature vectors as input, and an MLP is used to model the material;

[0041] For the design of the loss function, the albedo of surface points of the same semantic class is constrained to be the same according to the following formula, encouraging the phenomenon of the same material but different rendering colors to be explained by lighting and geometry:

[0042]

[0043] Where B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

[0044] In combination with the first aspect, in one embodiment, after obtaining and After that, it also includes:

[0045] Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters;

[0046] Using a smooth loss function for the albedo, we finally get the total loss function l of the inverse rendering process:

[0047] l=λ recon l recon +λ f l f +λ in l in +λ shade l cato

[0048] Among them, λ recon , f , in , shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

[0049] In a second aspect, an embodiment of the present application provides an inverse rendering device for fusion semantic segmentation considering shadows, and the inverse rendering device for fusion semantic segmentation considering shadows includes:

[0050] A reconstruction module, which is used to reconstruct the geometry of the scene, i.e., the signed distance function of the scene, using a set of multi-view photos of the 3D scene as input to achieve neural implicit surface reconstruction;

[0051] A projection module, which is used to take the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtain a surface point set with albedo semantic categories of the scene, and project the surface point set to the image plane to obtain a semantic segmentation map of the albedo of each view image of the scene;

[0052] A module is used for sampling rays on surface points of a scene using a set of multi-view photos of the three-dimensional scene and the obtained semantic segmentation map as input;

[0053] The rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

[0054] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0055] It can effectively deal with the ill-posed problem of inverse rendering, and more accurately decompose the geometry, material and lighting of the scene, by first reconstructing the signed distance function of the scene, and on this basis, obtaining the surface point set corresponding to the semantic segmentation map of the sparse perspective image of the scene, and further projecting it to the image plane to obtain the semantic segmentation map of each perspective image of the scene, thereby realizing the semantic segmentation of multi-perspective images of three-dimensional scenes under sparse supervision. This process does not require additional network training. Finally, considering that the number of materials in the scene is certain and limited, the results of semantic segmentation are further used to explicitly constrain material prediction. When there are obvious shadows in the reconstructed scene, the invention of the present application can effectively reduce the influence of shadows on material estimation, thereby achieving more accurate estimation of lighting and material. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of an inverse rendering method for fusion semantic segmentation considering shadows for this application;

[0057] Figure 2 A schematic diagram of the functional modules of the inverse rendering device for fusion semantic segmentation taking shadows into consideration in this application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0060] In a first aspect, an embodiment of the present application provides an inverse rendering method that integrates semantic segmentation and takes shadows into consideration.

[0061] In one embodiment, referring to Figure 1 , Figure 1 The following is a flowchart of the inverse rendering method of the fusion semantic segmentation considering shadows in this application. Figure 1 As shown, the inverse rendering method of fusion semantic segmentation considering shadows includes:

[0062] S1: Using a set of multi-view photos of a 3D scene as input to reconstruct the scene’s geometry, i.e., the signed distance function of the scene, to achieve neural implicit surface reconstruction;

[0063] Furthermore, in one embodiment, the geometric reconstruction of the three-dimensional scene specifically includes:

[0064] S101: A pixel on the image and the camera form a ray, and a finite number of spatial discrete points are sampled along the camera ray, so as to approximate the integral value of the volume rendering integral equation;

[0065] Among them, the volume rendering integral equation is specifically expressed as:

[0066]

[0067] Among them, C 1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c i Represents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

[0068] S102: Two multi-layer perceptrons (MLP) are used to map the spatial position to the symbol distance, geometric feature vector and color respectively;

[0069] S103: Execute forward rendering, use the signed distance function as the weight of the volume rendering integral, and optimize the signed distance function by minimizing the reconstruction loss of the rendered color and the observed color.

[0070] Specifically, in the actual application process, for a certain pixel on the image, under the premise of knowing the camera position, camera intrinsic parameter matrix and camera extrinsic parameter matrix, a camera ray is generated, and 128 spatial points are sampled on the ray, and the spatial position is encoded, and then the position encoding is input into MLPs (multi-layer perceptron) to obtain the signed distance and geometric feature vector of the spatial point. MLPs contains 8 hidden layers, each with 256 hidden units. Then the value of the volume rendering integral equation is calculated, and the network parameters are continuously optimized during network training by minimizing the reconstruction error between the volume rendering integral equation and the true color value of the pixel.

[0071] S2: Taking the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtain the surface point set of the scene with the albedo semantic category, and project the surface point set to the image plane to obtain the semantic segmentation map of the albedo of each view image of the scene; that is, realize the semantic segmentation of the multi-view image of the three-dimensional scene based on the neural radiation field;

[0072] Furthermore, in one embodiment, a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene is used as input to obtain a surface point set with an albedo semantic category of the scene, and the surface point set is projected onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene, specifically including:

[0073] S201: selecting albedo categories of sparse view images of a three-dimensional scene for annotation, and the selected sparse view images need to cover all positions of the three-dimensional scene, so as to obtain a semantic segmentation map I of the albedo of the corresponding view image N ;

[0074] S202: Input the obtained semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we obtain the point x on the ray where the signed distance function value is 0 i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thereby obtaining a surface point set X = {x|x = Ω(p), p∈I N}, where x, Ω(p) represent surface points and p represents pixel points;

[0075] S203: Projecting the surface points in the surface point set onto the image plane, thereby obtaining a semantic segmentation map of the albedo of the image at each viewing angle of the scene;

[0076] S204: De-noising the obtained semantic segmentation map by using a median filter to obtain a final semantic segmentation map. That is, considering the calculation error of the ray tracing process, the obtained semantic segmentation map of the scene will have more noise points, so the obtained semantic segmentation map is de-noised by using a median filter.

[0077] In the present application, the surface points in the surface point set are projected onto the image plane, wherein the specific process of projection includes:

[0078] S2031: The viewing angle image I to be projected is known j The intrinsic parameter matrix K of the corresponding camera jand the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system;

[0079] S2032: For surface point x i , which is projected onto I j The pixel point p on j (u j ,v j )=K j ·T j ·x i , thus establishing p i (u i , v i ) and p j (u j ,v j ), that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

[0080] S3: Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene;

[0081] S4: Forward rendering is performed based on the incident light on the sampled light and the material of the surface point. The lighting and material are optimized through network training, and the accurate lighting and material are decomposed. That is, inverse rendering under semantic constraints is achieved.

[0082] Furthermore, in one embodiment, forward rendering is performed based on the incident light on the sampling light and the material of the surface point, wherein the inverse rendering of the three-dimensional scene specifically includes:

[0083] S401: Modeling the lighting and material of the scene based on the signed distance function of the scene and performing forward rendering;

[0084] S402: Optimize lighting and materials by minimizing reconstruction loss of rendered color and observed color;

[0085] Among them, for forward rendering, forward rendering uses a physically based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as:

[0086]

[0087] in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function is Ω represents a hemisphere centered at the surface point normal vector.

[0088] Further, in this application, for The specific calculation method is: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting, wherein the direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene;

[0089] For sampling light, multiple importance sampling technology is used, which combines the advantages of multiple sampling methods;

[0090] For ambient light, a spherical Gaussian function is used for modeling, specifically:

[0091]

[0092] Among them, E(w i ) indicates w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, M represents the number of spherical Gaussian functions;

[0093] For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP (multi-layer perceptron) is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i Because the outgoing radiance of a scene surface point is equal to the observed color of the corresponding camera ray, indirect lighting is optimized by minimizing the difference between the outgoing radiance and the true value.

[0094] Specifically, when the scene geometry is known, the scene's lighting and materials are further modeled. On the one hand, a three-dimensional scene receives ambient light from the outside, i.e., direct lighting, and on the other hand, it also receives indirect lighting from the inside of the scene due to specular reflection and diffuse reflection. For ambient light, a spherical Gaussian function is used for modeling; for indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP is used to model the indirect lighting.

[0095] The propagation process of light inside the scene is extremely complex, and it is impractical to trace the propagation path of light inside the scene forward. For the total illumination received by the scene surface, light is sampled directly from the surface point of the scene and on a hemisphere centered on the surface point normal vector, and the incident light in the direction of the sampled light is directly queried, including direct illumination and indirect illumination. For light sampling, Multiple Importance Sampling (MIS) is used. MIS can combine the advantages of multiple sampling methods. The light sampling method used in this embodiment specifically includes cosine importance sampling and SG distribution sampling.

[0096] In this application, The specific calculation process is:

[0097] S11: For the input of the material network model, the albedo semantic category is encoded using one-hot encoding and combined with surface points and geometric feature vectors as input, and an MLP is used to model the material;

[0098] S12: For the design of the loss function, the albedo of surface points of the same semantic class is constrained to be the same according to the following formula, and the phenomenon of different rendering colors with the same material is encouraged to be explained by lighting and geometry:

[0099]

[0100] Where B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

[0101] The obtained semantic segmentation map is used to constrain the prediction of materials; that is, considering that the number of materials in the scene is certain and limited, the semantic segmentation map obtained above is used to constrain the prediction of materials, which is specifically reflected in the input of the material network model and the design of the loss function.

[0102] Further, in getting and After that, it also includes:

[0103] S21: Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters;

[0104] S22: Use a smooth loss function for the albedo, and finally get the total loss function l of the inverse rendering process:

[0105] l=λ recon l recon +λ f l f +λ in l in +λ shade l cato

[0106] Among them, λ recon , f , in , shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

[0107] The inverse rendering method of the embodiment of the present application that considers the fusion semantic segmentation of shadows can effectively deal with the ill-posed problem of inverse rendering, and more accurately decompose the geometry, material and lighting of the scene, by first reconstructing the signed distance function of the scene, and on this basis, obtaining the surface point set corresponding to the semantic segmentation map of the sparse perspective image of the scene, and further projecting it to the image plane to obtain the semantic segmentation map of each perspective image of the scene, thereby realizing the semantic segmentation of multi-perspective images of the three-dimensional scene under sparse supervision, and this process does not require additional network training. Finally, considering that the number of materials in the scene is certain and limited, the results of semantic segmentation are further used to explicitly constrain material prediction. When there are obvious shadows in the reconstructed scene, the invention of the present application can effectively reduce the influence of shadows on material estimation, thereby achieving more accurate estimation of lighting and material.

[0108] In a second aspect, an embodiment of the present application also provides an inverse rendering device that takes shadow-fused semantic segmentation into consideration.

[0109] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of the inverse rendering device for considering the fusion semantic segmentation of shadows in this application. Figure 2 As shown, the inverse rendering device of fusion semantic segmentation considering shadows includes: a reconstruction module, a projection module, an adoption module, and a rendering module.

[0110] The reconstruction module is used to reconstruct the geometric shape of the scene, that is, the signed distance function of the scene, using a set of multi-view photos of the three-dimensional scene as input, to realize neural implicit surface reconstruction; the projection module is used to take the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtain the surface point set of the scene with the semantic category of the albedo, and project the surface point set to the image plane to obtain the semantic segmentation map of the albedo of each view image of the scene; the sampling module is used to take a set of multi-view photos of the three-dimensional scene and the obtained semantic segmentation map as input, and perform light sampling on the surface points of the scene; the rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point, optimize the lighting and material through network training, and decompose to obtain accurate lighting and material.

[0111] Among them, the functional implementation of each module in the above-mentioned inverse rendering device of fused semantic segmentation considering shadows corresponds to the steps in the above-mentioned inverse rendering method embodiment of fused semantic segmentation considering shadows, and its functions and implementation processes are no longer repeated here one by one.

[0112] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0113] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0114] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0115] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0117] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An inverse rendering method for fusion semantic segmentation considering shadows, characterized in that: The inverse rendering method of fusion semantic segmentation considering shadows includes: Taking a set of multi-view photos of a 3D scene as input, the geometric shape of the scene, i.e. the signed distance function of the scene, is reconstructed to achieve neural implicit surface reconstruction; Taking the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, a surface point set with the albedo semantic category of the scene is obtained, and the surface point set is projected onto the image plane to obtain the semantic segmentation map of the albedo of the scene at each view image; Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene; Forward rendering is performed based on the incident light on the sampling light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

2. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 1, characterized in that: The geometric reconstruction of the 3D scene includes: A pixel on the image forms a ray with the camera, and a finite number of spatial discrete points are sampled along the camera ray to approximate the integral value of the volume rendering integral equation; Two multi-layer perceptrons are used to map spatial positions to symbolic distance, geometric feature vectors and colors respectively; Perform forward rendering and use the signed distance function as the weight of the volume rendering integral. The signed distance function is optimized by minimizing the reconstruction loss of the rendered color and the observed color.

3. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 2, characterized in that: The volume rendering integral equation is specifically expressed as: Where C1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c i Represents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

4. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 1, characterized in that: The method uses the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtains a surface point set with the albedo semantic category of the scene, and projects the surface point set to the image plane to obtain the semantic segmentation map of the albedo of each view image of the scene, specifically including: The albedo categories of the sparse view images of the three-dimensional scene are selected for annotation, and the selected sparse view images need to cover all positions of the three-dimensional scene, so as to obtain the semantic segmentation map I of the albedo of the corresponding view image N ; Input semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we obtain the point x on the ray where the signed distance function value is 0 i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thus obtaining a surface point set X = {xx = Ω(p), p∈I N }, where x, Ω(p) represent surface points and p represents pixel points; Project the surface points in the surface point set onto the image plane to obtain a semantic segmentation map of the albedo of each viewpoint of the scene; The obtained semantic segmentation map is denoised using median filtering to obtain the final semantic segmentation map.

5. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 4, characterized in that: The surface points in the surface point set are projected onto the image plane, wherein the specific process of projection includes: Known view image I to be projected j The intrinsic parameter matrix K of the corresponding camera j and the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system; For a surface point x i , which is projected onto I j The pixel point p on j (u j ,v j )=K j ·T j ·x i , thus establishing p i (u i ,v i ) and p j (u j ,v j ), that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

6. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 1, characterized in that: The forward rendering is performed based on the incident light on the sampling light and the material of the surface point, wherein the inverse rendering of the three-dimensional scene specifically includes: Based on the scene's signed distance function, the scene's lighting and materials are modeled and forward rendering is performed; Optimize lighting and materials by minimizing reconstruction loss of rendered and observed colors; Among them, for forward rendering, forward rendering uses a physically based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as: in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function is Ω represents a hemisphere centered at the surface point normal vector.

7. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 6, characterized in that: for The specific calculation method is: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting, wherein the direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene; For sampling rays, multiple importance sampling technique is used; For ambient light, a spherical Gaussian function is used for modeling. Specifically: Among them, E(w i ) indicates w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, M represents the number of spherical Gaussian functions; For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i The outgoing radiance in the direction.

8. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 7, characterized in that: for The specific calculation process is: For the input of the material network model, the albedo semantic category is encoded using one-hot encoding and combined with surface points and geometric feature vectors as input, and an MLP is used to model the material; For the design of the loss function, the albedo of surface points of the same semantic class is constrained to be the same according to the following formula, encouraging the phenomenon of the same material but different rendering colors to be explained by lighting and geometry: Where B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

9. The inverse rendering method of fusion semantic segmentation considering shadows as claimed in claim 8, characterized in that: In getting and After that, it also includes: Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters; Using a smooth loss function for the albedo, we finally get the total loss function l of the inverse rendering process: l=λ recon l recon +λ f l f +λ in l in +λ shade l cato Among them, λ recon , f , in , shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

10. An inverse rendering device for fusion semantic segmentation considering shadows, characterized in that: The inverse rendering device of the fusion semantic segmentation considering shadows comprises: A reconstruction module, which is used to reconstruct the geometry of the scene, i.e., the signed distance function of the scene, using a set of multi-view photos of the 3D scene as input to achieve neural implicit surface reconstruction; A projection module, which is used to take the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtain a surface point set with albedo semantic categories of the scene, and project the surface point set to the image plane to obtain a semantic segmentation map of the albedo of each view image of the scene; A module is used for sampling rays on surface points of a scene using a set of multi-view photos of the three-dimensional scene and the obtained semantic segmentation map as input; The rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

Images