Inverse rendering method and device considering shadow fusion semantic segmentation

Abstract

The present invention discloses an inverse rendering method and device for fused semantic segmentation that considers shadows, relating to the fields of computer vision and graphics. The method comprises: using a set of multi-view photos of a three-dimensional scene as input to reconstruct the geometric shape of the scene, thereby realizing neural implicit surface reconstruction; using a semantic segmentation map of the albedo of the sparse view images of the three-dimensional scene as input to obtain a surface point set of the scene with albedo semantic categories, and projecting the surface point set onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene; using a set of multi-view photos of the three-dimensional scene and the obtained semantic segmentation map as input to perform light sampling on the surface points of the scene; performing forward rendering based on the incident light on the sampled light and the material of the surface points, optimizing the lighting and material through network training, and decomposing to obtain accurate lighting and material. The present application can effectively reduce the impact of shadows on material estimation, thereby achieving relatively accurate lighting and material estimation.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and graphics, and in particular to an inverse rendering method and device for fusion semantic segmentation taking shadows into consideration. Background Art

[0002] Using two-dimensional images taken from the real world to reconstruct high-fidelity three-dimensional scenes in a computer can be applied to virtual reality, augmented reality, and digital twins. However, two-dimensional images correspond to a specific state of the real scene. Considering more flexible applications, such as the ability to modify the scene, including relighting and material editing, it is necessary to obtain the material and lighting properties of the real scene. This work is usually called inverse rendering, which refers to recovering the physical properties of the three-dimensional scene, such as the geometry, material properties, and lighting environment, from the two-dimensional image. However, there is a great deal of uncertainty in correctly decomposing these properties from the result of the joint synthesis of geometry, material, and lighting. Different combinations of these properties may have the same synthesis result. Due to its complexity and ill-posedness, this work has always been a challenging problem in the field of computer vision and graphics.

[0003] Early work on inverse rendering focused on intrinsic image decomposition, decomposing images into albedo and brightness maps. This can be broadly categorized into two types: those based on manually set prior knowledge and those based on deep learning. Methods based on manually set prior knowledge often use empirical knowledge to set thresholds to decompose the albedo and brightness of an image; deep learning-based methods typically require albedo label values ​​for supervision, but the albedo of real scenes is often difficult to obtain. Current implicit neural rendering methods have achieved good results in new view synthesis and 3D reconstruction, and their emergence has also brought new solutions to inverse rendering. Current research focuses on learning the geometry and materials of scenes using multilayer perceptrons (MLPs), and using physically based rendering equations to replace the volume rendering equations in Neural Radiance Fields (NeRFs) to address this issue.

[0004] However, in inverse rendering, the presence of shadows can interfere with material estimation, making it easy to interpret shadows as material properties of the scene itself. Current work on implicit neural rendering has not directly addressed this issue. Additionally, some studies have leveraged prior knowledge, such as using semantic classification results obtained from a semantic segmentation network to directly constrain regions belonging to the same category to have the same material. While this approach mitigates the impact of shadows on material prediction, the 2D semantic segmentation network employed requires semantic segmentation label values ​​for a large number of viewpoints and has weak generalization capabilities, making it impossible to obtain relatively accurate semantic segmentation results for each viewpoint of a 3D scene. Summary of the Invention

[0005] The present application provides an inverse rendering method and device that considers shadow fusion semantic segmentation, which can effectively reduce the impact of shadows on material estimation, thereby achieving more accurate lighting and material estimation.

[0006] In a first aspect, an embodiment of the present application provides an inverse rendering method for fusion semantic segmentation taking into account shadows, the inverse rendering method for fusion semantic segmentation taking into account shadows comprising:

[0007] Taking a set of multi-view photos of a 3D scene as input, the geometric shape of the scene is reconstructed, i.e., the signed distance function of the scene, to achieve neural implicit surface reconstruction;

[0008] Taking the semantic segmentation map of the albedo of the sparse view image of the 3D scene as input, a surface point set with albedo semantic categories of the scene is obtained, and the surface point set is projected onto the image plane to obtain the semantic segmentation map of the albedo of the scene image at each view point.

[0009] Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene;

[0010] Forward rendering is performed based on the incident light on the sampling light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

[0011] In conjunction with the first aspect, in one embodiment, geometric reconstruction of a three-dimensional scene specifically includes:

[0012] A pixel on the image is combined with the camera to form a ray, and a finite number of spatial discrete points are sampled along the camera ray to approximate the integral value of the volume rendering integral equation;

[0013] Two multi-layer perceptrons are used to map spatial positions to symbolic distance, geometric feature vectors, and colors respectively;

[0014] Perform forward rendering, use the signed distance function as the weight of the volume rendering integral, and optimize the signed distance function by minimizing the reconstruction loss of rendered color and observed color.

[0015] In combination with the first aspect, in one implementation, the volume rendering integral equation is specifically expressed as:

[0016]

[0017] Where C1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c i Represents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

[0018] In conjunction with the first aspect, in one embodiment, the method of using a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene as input, obtaining a surface point set of the scene with albedo semantic categories, and projecting the surface point set onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene specifically includes:

[0019] The albedo categories of the sparse view images of the three-dimensional scene are selected for annotation, and the selected sparse view images need to cover all positions of the three-dimensional scene, so as to obtain the semantic segmentation map I of the albedo of the corresponding view image N ;

[0020] Input semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we can get the point x on the ray where the signed distance function value is 0. i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thus obtaining the surface point set X={xx=Ω(p),p∈I N}, where x, Ω(p) represent surface points and p represents pixel points;

[0021] Project the surface points in the surface point set onto the image plane to obtain a semantic segmentation map of the albedo of the image at each viewpoint of the scene;

[0022] The obtained semantic segmentation map is denoised using median filtering to obtain the final semantic segmentation map.

[0023] In combination with the first aspect, in one embodiment, the surface points in the surface point set are projected onto the image plane, wherein the specific process of the projection includes:

[0024] The viewing angle image I to be projected is known j The intrinsic parameter matrix K of the corresponding camera j and the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system;

[0025] For a surface point x i , which is projected onto I j Pixel point p on j (uj ,v j )=K j ·T j ·x i , thus establishing p i (u i ,v i ) and p j (u j ,v j ) mapping relationship, that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

[0026] In combination with the first aspect, in one embodiment, forward rendering is performed based on the incident light on the sampling ray and the material of the surface point, wherein the inverse rendering of the three-dimensional scene specifically includes:

[0027] Based on the scene's signed distance function, the scene's lighting and materials are modeled and forward rendering is performed;

[0028] Optimize lighting and materials by minimizing the reconstruction loss of rendered and observed colors;

[0029] Among them, for forward rendering, forward rendering uses a physics-based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as:

[0030]

[0031] in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function, Ω represents a hemisphere centered at the surface point normal vector.

[0032] In conjunction with the first aspect, in one embodiment,

[0033] for The specific calculation method is as follows: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, and querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting. The direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene.

[0034] For sampling rays, multiple importance sampling technique is used;

[0035] For ambient light, a spherical Gaussian function is used for modeling. Specifically:

[0036]

[0037] Among them, E(w i ) represents w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, and M represents the number of spherical Gaussian functions;

[0038] For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i The outgoing radiance in the direction.

[0039] In combination with the first aspect, in one embodiment, for The specific calculation process is:

[0040] For the input of the material network model, the albedo semantic category is encoded using a one-hot encoding method and combined with the surface points and geometric feature vectors as input, and an MLP is used to model the material;

[0041] For the design of the loss function, the following formula is used to constrain the albedo of surface points of the same semantic class to be the same, and to encourage the phenomenon of different rendering colors with the same material to be explained by lighting and geometry:

[0042]

[0043] Among them, B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

[0044] In combination with the first aspect, in one embodiment, after obtaining and After that, it also includes:

[0045] Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters;

[0046] Using a smooth loss function for the albedo, we finally get the total loss function l of the inverse rendering process:

[0047] l=λ recon l recon +λ f l f +λ in l in +λ shade l cato

[0048] Among them, λ recon ,λ f ,λ in ,λ shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

[0049] In a second aspect, an embodiment of the present application provides an inverse rendering device for fusion semantic segmentation taking into account shadows, the inverse rendering device for fusion semantic segmentation taking into account shadows comprising:

[0050] A reconstruction module, which is used to reconstruct the geometry of the scene, i.e., the signed distance function of the scene, using a set of multi-view photos of the 3D scene as input to implement neural implicit surface reconstruction;

[0051] A projection module is configured to take a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene as input, obtain a surface point set of the scene with albedo semantic categories, and project the surface point set onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene;

[0052] A module is used for sampling rays on surface points of a 3D scene using a set of multi-view photos of the scene and the obtained semantic segmentation map as input;

[0053] The rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point. It optimizes the lighting and material through network training and decomposes them to obtain accurate lighting and material.

[0054] The beneficial effects of the technical solutions provided in the embodiments of the present application include:

[0055] It can effectively deal with the ill-posedness problem of inverse rendering and more accurately decompose the geometry, material and lighting of the scene by first reconstructing the signed distance function of the scene. On this basis, the surface point set corresponding to the semantic segmentation map of the sparse perspective image of the scene is obtained, and further projected to the image plane to obtain the semantic segmentation map of each perspective image of the scene, thereby realizing the semantic segmentation of multi-perspective images of the three-dimensional scene under sparse supervision. This process does not require additional network training. Finally, considering that the number of materials in the scene is certain and limited, the results of semantic segmentation are further used to explicitly constrain material prediction. When there are obvious shadows in the reconstructed scene, the invention of the present application can effectively reduce the influence of shadows on material estimation, thereby achieving more accurate lighting and material estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of the inverse rendering method for fusion semantic segmentation considering shadows in this application;

[0057] Figure 2 This is a schematic diagram of the functional modules of the inverse rendering device that considers shadow fusion semantic segmentation in this application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0060] In a first aspect, an embodiment of the present application provides an inverse rendering method that integrates semantic segmentation and takes shadows into consideration.

[0061] In one embodiment, referring to Figure 1 , Figure 1 This is a flowchart of the inverse rendering method of the fusion semantic segmentation considering shadows in this application. Figure 1 As shown in Figure 2, the inverse rendering method of fusion semantic segmentation considering shadows includes:

[0062] S1: Using a set of multi-view photos of a 3D scene as input, we reconstruct the scene’s geometry, i.e., the signed distance function of the scene, to achieve neural implicit surface reconstruction.

[0063] Furthermore, in one embodiment, geometric reconstruction of a three-dimensional scene specifically includes:

[0064] S101: A pixel on the image and the camera form a ray, and a finite number of spatial discrete points are sampled along the camera ray to approximate the integral value of the volume rendering integral equation;

[0065] Among them, the volume rendering integral equation is specifically expressed as:

[0066]

[0067] Where C1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c i Represents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

[0068] S102: Two multi-layer perceptrons (MLPs) are used to map spatial positions to symbolic distances, geometric feature vectors, and colors, respectively.

[0069] S103: Execute forward rendering, use the signed distance function as a weight for volume rendering integral, and optimize the signed distance function by minimizing the reconstruction loss of rendered color and observed color.

[0070] Specifically, in actual applications, for a pixel on an image, given the known camera position, camera intrinsic parameter matrix, and camera extrinsic parameter matrix, a camera ray is generated. 128 spatial points are sampled along the ray and position-encoded. This position code is then fed into an MLP (Multi-Layer Perceptron) to obtain the signed distance and geometric feature vector of the spatial point. The MLP consists of eight hidden layers, each with 256 hidden units. The volume rendering integral equation is then calculated, and network parameters are continuously optimized during network training by minimizing the reconstruction error between the volume rendering integral equation and the true pixel color value.

[0071] S2: Using the semantic segmentation map of the albedo of the sparse view images of the 3D scene as input, a surface point set with albedo semantic categories is obtained for the scene. The surface point set is projected onto the image plane to obtain a semantic segmentation map of the albedo of the scene images at each viewpoint; that is, semantic segmentation of the 3D scene multi-view images based on the neural radiance field is achieved.

[0072] Furthermore, in one embodiment, a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene is used as input to obtain a surface point set of the scene with albedo semantic categories, and the surface point set is projected onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene, specifically including:

[0073] S201: Select the albedo category of the sparse view images of the three-dimensional scene for labeling, and the selected sparse view images must cover all positions of the three-dimensional scene, so as to obtain the semantic segmentation map I of the albedo of the corresponding view images N ;

[0074] S202: Input the obtained semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we can get the point x on the ray where the signed distance function value is 0. i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thereby obtaining a surface point set X = {x|x = Ω(p), p∈I N}, where x, Ω(p) represent surface points and p represents pixel points;

[0075] S203: Projecting the surface points in the surface point set onto the image plane, thereby obtaining a semantic segmentation map of the albedo of the image at each viewing angle of the scene;

[0076] S204: De-noising the obtained semantic segmentation map using a median filter to obtain a final semantic segmentation map. That is, considering the computational error in the ray tracing process, the obtained semantic segmentation map of the scene will contain a large number of noise points, so the median filter is used to de-noise the obtained semantic segmentation map.

[0077] In this application, the surface points in the surface point set are projected onto the image plane, wherein the specific process of projection includes:

[0078] S2031: The viewing angle image I to be projected is known j The intrinsic parameter matrix K of the corresponding camera j and the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system;

[0079] S2032: For surface point xi , which is projected onto I j Pixel point p on j (u j ,v j )=K j ·T j ·x i , thus establishing p i (u i , v i ) and p j (u j ,v j ) mapping relationship, that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

[0080] S3: Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene;

[0081] S4: Forward rendering is performed based on the incident light on the sampled rays and the material of the surface points. Through network training, the lighting and materials are optimized and decomposed to obtain accurate lighting and materials. This is inverse rendering under semantic constraints.

[0082] Furthermore, in one embodiment, forward rendering is performed based on the incident light on the sampling ray and the material of the surface point. Specifically, for the inverse rendering of the three-dimensional scene, the specific process includes:

[0083] S401: Modeling the lighting and material of the scene based on the signed distance function of the scene and performing forward rendering;

[0084] S402: Optimize lighting and materials by minimizing the reconstruction loss of rendered color and observed color;

[0085] Among them, for forward rendering, forward rendering uses a physics-based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as:

[0086]

[0087] in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function, Ω represents a hemisphere centered at the surface point normal vector.

[0088] Furthermore, in this application, The specific calculation method is as follows: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, and querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting. The direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene.

[0089] For sampling light, multiple importance sampling technology is used, which combines the advantages of multiple sampling methods;

[0090] For ambient light, a spherical Gaussian function is used for modeling. Specifically:

[0091]

[0092] Among them, E(w i ) represents w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, and M represents the number of spherical Gaussian functions;

[0093] For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light and the scene at the surface point, and an MLP (multi-layer perceptron) is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i Because the outgoing radiance of a surface point in the scene is equal to the observed color of the corresponding camera ray, indirect lighting is optimized by minimizing the difference between the outgoing radiance and the true value.

[0094] Specifically, given the known scene geometry, the scene's lighting and materials are further modeled. A 3D scene receives both external ambient light (direct illumination) and indirect illumination from internal specular and diffuse reflections. Ambient light is modeled using a spherical Gaussian function. Indirect lighting is simplified to the outgoing radiance at the intersection of a sampled ray from a surface point and the scene, and is modeled using an MLP.

[0095] The propagation process of light within the scene is extremely complex, and it is impractical to trace the propagation path of light within the scene forward. For the total illumination received by the scene surface, we start directly from the surface point of the scene and sample the light on the hemisphere centered on the surface point normal vector, and directly query the incident light in the direction of the sampled light, including direct illumination and indirect illumination. For light sampling, the Multiple Importance Sampling (MIS) technique is used. MIS can combine the advantages of multiple sampling methods. The light sampling method used in this embodiment specifically includes cosine importance sampling and SG distribution sampling.

[0096] In this application, The specific calculation process is:

[0097] S11: For the input of the material network model, the albedo semantic category is encoded using a one-hot encoding method and combined with the surface points and geometric feature vectors as input, and an MLP is used to model the material;

[0098] S12: For the design of the loss function, the following formula is used to constrain the albedo of surface points of the same semantic class to be the same, and to encourage the phenomenon of different rendering colors with the same material to be explained by lighting and geometry:

[0099]

[0100] Among them, B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

[0101] The obtained semantic segmentation map is used to constrain the prediction of materials; that is, considering that the number of materials in the scene is certain and limited, the semantic segmentation map obtained above is used to constrain the prediction of materials, which is specifically reflected in the input of the material network model and the design of the loss function.

[0102] Further, in getting and After that, it also includes:

[0103] S21: Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters;

[0104] S22: Use a smooth loss function for the albedo, and finally get the total loss function l of the inverse rendering process:

[0105] l=λ recon l recon +λ f l f +λ in l in +λ shade l cato

[0106] Among them, λ recon ,λ f ,λ in ,λ shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

[0107] The inverse rendering method of the embodiment of the present application that considers shadow fusion semantic segmentation can effectively deal with the ill-posedness problem of inverse rendering, and more accurately decompose the geometry, material and lighting of the scene. By first reconstructing the signed distance function of the scene, and on this basis, obtaining the surface point set corresponding to the semantic segmentation map of the sparse perspective image of the scene, and further projecting it to the image plane to obtain the semantic segmentation map of each perspective image of the scene, thereby realizing the semantic segmentation of multi-perspective images of the three-dimensional scene under sparse supervision. This process does not require additional network training. Finally, considering that the number of materials in the scene is certain and limited, the results of semantic segmentation are further used to explicitly constrain material prediction. When there are obvious shadows in the reconstructed scene, the invention of the present application can effectively reduce the influence of shadows on material estimation, thereby achieving more accurate lighting and material estimation.

[0108] In a second aspect, an embodiment of the present application further provides an inverse rendering device that considers shadow-fused semantic segmentation.

[0109] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of the inverse rendering device that considers the fusion semantic segmentation of shadows in this application. Figure 2 As shown, the inverse rendering device of fusion semantic segmentation considering shadows includes: a reconstruction module, a projection module, an adoption module, and a rendering module.

[0110] The reconstruction module is used to reconstruct the geometric shape of the scene, that is, the signed distance function of the scene, using a set of multi-view photos of the three-dimensional scene as input, to realize neural implicit surface reconstruction; the projection module is used to take the semantic segmentation map of the albedo of the sparse view image of the three-dimensional scene as input, obtain the surface point set of the scene with albedo semantic categories, and project the surface point set to the image plane to obtain the semantic segmentation map of the albedo of each view image of the scene; the adoption module is used to take a set of multi-view photos of the three-dimensional scene and the obtained semantic segmentation map as input, and perform light sampling on the surface points of the scene; the rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point, optimize the lighting and material through network training, and decompose to obtain accurate lighting and material.

[0111] Among them, the functional implementation of each module in the above-mentioned inverse rendering device of fused semantic segmentation considering shadows corresponds to the steps in the above-mentioned inverse rendering method embodiment of fused semantic segmentation considering shadows, and their functions and implementation processes are no longer repeated here.

[0112] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0113] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0114] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0115] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of the present application.

[0117] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A shadow-considered fusion semantic segmentation inverse rendering method, characterized in that: The inverse rendering method of fusion semantic segmentation considering shadows includes: Taking a set of multi-view photos of a 3D scene as input, the geometric shape of the scene is reconstructed, i.e., the signed distance function of the scene, to achieve neural implicit surface reconstruction; Taking the semantic segmentation map of the albedo of the sparse view image of the 3D scene as input, a surface point set with albedo semantic categories of the scene is obtained, and the surface point set is projected onto the image plane to obtain the semantic segmentation map of the albedo of the scene image at each view point. Taking a set of multi-view photos of a 3D scene and the obtained semantic segmentation map as input, light sampling is performed on the surface points of the scene; Forward rendering is performed based on the incident light on the sampling light and the material of the surface point. The lighting and material are optimized through network training to decompose and obtain accurate lighting and material.

2. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 1, characterized in that: The geometric reconstruction of the 3D scene includes: A pixel on the image is combined with the camera to form a ray, and a finite number of spatial discrete points are sampled along the camera ray to approximate the integral value of the volume rendering integral equation; Two multi-layer perceptrons are used to map spatial positions to symbolic distance, geometric feature vectors, and colors respectively; Perform forward rendering, use the signed distance function as the weight of the volume rendering integral, and optimize the signed distance function by minimizing the reconstruction loss of rendered color and observed color.

3. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 2, characterized in that: The volume rendering integral equation is specifically expressed as: Where C1 represents the volume rendering integral equation, T i represents the cumulative transmittance, α i Indicates opacity, c i Represents the RGB color value of the spatial sampling point, and N represents the number of sampling points.

4. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 1, characterized in that: The method uses the semantic segmentation map of the albedo of the sparse view images of the three-dimensional scene as input, obtains a surface point set with albedo semantic categories of the scene, and projects the surface point set onto the image plane to obtain the semantic segmentation map of the albedo of the scene images at each view point, specifically including: The albedo categories of the sparse view images of the three-dimensional scene are selected for annotation, and the selected sparse view images need to cover all positions of the three-dimensional scene, so as to obtain the semantic segmentation map I of the albedo of the corresponding view image N ; Input semantic segmentation map I N , for the pixel point p i (u i ,v i ), the corresponding albedo category is A(p i ), by performing ray tracing on the corresponding camera ray, we can get the point x on the ray where the signed distance function value is 0. i =Ω(p i ), that is, the surface point of the scene, then its albedo category A(x i )=A(p i ), thus obtaining the surface point set X = {x|x = Ω(p), p∈I N }, where x, Ω(p) represent surface points and p represents pixel points; Project the surface points in the surface point set onto the image plane to obtain a semantic segmentation map of the albedo of the image at each viewpoint of the scene; The obtained semantic segmentation map is denoised using median filtering to obtain the final semantic segmentation map.

5. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 4, characterized in that: The surface points in the surface point set are projected onto the image plane, wherein the specific process of projection includes: The viewing angle image I to be projected is known j The intrinsic parameter matrix K of the corresponding camera j and the external parameter matrix T j , where K j Used to realize the transformation from camera coordinate system to image coordinate system, T j Used to realize the transformation from world coordinate system to camera coordinate system; For a surface point x i , which is projected onto I j Pixel point p on j (u j ,v j )=K j ·T j ·x i , thus establishing p i (u i ,v i ) and p j (u j ,v j ) mapping relationship, that is, p j =K j ·T j ·Ω(p i ), and A(p j )=A(p i ), A(p j ) is p j The corresponding albedo category.

6. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 1, characterized in that: The forward rendering is performed based on the incident light on the sampling light and the material of the surface point. The inverse rendering of the three-dimensional scene specifically includes: Based on the scene's signed distance function, the scene's lighting and materials are modeled and forward rendering is performed; Optimize lighting and materials by minimizing the reconstruction loss of rendered and observed colors; Among them, for forward rendering, forward rendering uses a physics-based rendering equation to calculate the surface point on the camera ray The rendering color of the direction, specifically expressed as: in, Indicates the rendering color, Representing surface points In w o The outgoing radiance in the direction, Representing surface points In w o The incident radiance in the direction, Representing surface points In the incident direction w i , the emission direction is w o The bidirectional reflectance distribution function, Ω represents a hemisphere centered at the surface point normal vector.

7. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 6, characterized in that: for The specific calculation method is as follows: starting from the surface point of the scene, sampling light on the hemisphere centered on the surface point normal vector, and querying the incident light in the direction of the sampled light, including direct lighting and indirect lighting. The direct lighting is the ambient light received by the scene from the outside, and the indirect lighting is the lighting formed by specular reflection and diffuse reflection inside the scene. For sampling rays, multiple importance sampling technique is used; For ambient light, a spherical Gaussian function is used for modeling. Specifically: Among them, E(w i ) represents w i Directional ambient light, ξ k represents the direction of the lobe, λ k Indicates the sharpness of the control lobe, μ k represents the amplitude of the lobe, and M represents the number of spherical Gaussian functions; For indirect lighting, the indirect lighting is simplified to the outgoing radiance of the intersection of the sampled light at the surface point and the scene, and an MLP is used to model the indirect lighting. In w i Directional incident radiance in, represents the intersection of the sampled ray and the scene, Representing surface points In w i Visibility to ambient light in a direction, Representing surface points In -w i The outgoing radiance in the direction.

8. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 7, characterized in that: for The specific calculation process is: For the input of the material network model, the albedo semantic category is encoded using a one-hot encoding method and combined with the surface points and geometric feature vectors as input, and an MLP is used to model the material; For the design of the loss function, the following formula is used to constrain the albedo of surface points of the same semantic class to be the same, and to encourage the phenomenon of different rendering colors with the same material to be explained by lighting and geometry: Where B represents the number of semantic categories, X b represents the surface point set of the bth semantic class, l cato represents the loss function, a x represents the albedo of the x-th surface point.

9. The inverse rendering method of fusion semantic segmentation considering shadows according to claim 8, characterized in that: In getting and After that, it also includes: Calculate physically based rendering colors By minimizing and observe the color The reconstruction loss To optimize both lighting and material network parameters; Using a smooth loss function for the albedo, we finally get the total loss function l of the inverse rendering process: l=λ recon l recon +λ f l f +λ in l in +λ shade l cato Among them, λ recon ,λ f ,λ in ,λ shade represents the weight, l in It means minimizing the difference between the outgoing radiance and the true value.

10. An inverse rendering device for fusion semantic segmentation considering shadows, characterized in that: The inverse rendering device of the fusion semantic segmentation considering shadows includes: A reconstruction module, which is used to reconstruct the geometry of the scene, i.e., the signed distance function of the scene, using a set of multi-view photos of the 3D scene as input to implement neural implicit surface reconstruction; A projection module is configured to take a semantic segmentation map of the albedo of a sparse view image of a three-dimensional scene as input, obtain a surface point set of the scene with albedo semantic categories, and project the surface point set onto an image plane to obtain a semantic segmentation map of the albedo of each view image of the scene; A module is used for sampling rays on surface points of a 3D scene using a set of multi-view photos of the scene and the obtained semantic segmentation map as input; The rendering module is used to perform forward rendering based on the incident light on the sampled light and the material of the surface point. It optimizes the lighting and material through network training and decomposes them to obtain accurate lighting and material.

Images