A novel editable view synthesis method based on intrinsic neural radiance field

Abstract

The present invention provides an editable novel view synthesis method based on intrinsic neural radiance fields. The present invention adopts the representation method of intrinsic neural radiance fields, IntrinsicNeRF, and introduces the intrinsic image decomposition technology into the neural rendering method NeRF for the first time, which can handle scenes of unlimited scale. In order to meet the challenge of the under-constrained inverse problem of eigendecomposition, the present invention adopts a distance-aware point sampling method and an iterative clustering optimization method, introduces traditional intrinsic decomposition priors to perform unsupervised training on IntrinsicNeRF, and thus obtains temporally and spatially consistent eigendecomposition results. In order to prevent similar reflectivities of different objects in a larger scene from being incorrectly clustered together, the present invention adopts a hierarchical clustering method based on semantic information. The present invention can perform real-time color editing, material editing, and lighting changes on video scenes. Blender Objects and Replica Scene experiments show that IntrinsicNeRF can obtain consistent intrinsic decomposition results and high-fidelity novel view synthesis.

Description

Technical Field

[0001] The present invention relates to the fields of intrinsic image decomposition and neural rendering, and in particular to an editable novel view synthesis method based on intrinsic neural radiance fields (Intrinsic NeRF). Background Art

[0002] Intrinsic image decomposition involves decomposing an image into two layers: reflectance (R) and shading (S). R represents the material color of the object itself, and S represents the brightness variations in the image caused by varying illumination distributions and geometric variations. The results of intrinsic image decomposition can be used to achieve realistic texture and lighting effects, offering broad application prospects and, therefore, has been extensively studied. However, this problem is extremely difficult. Theoretically, there are countless possible solutions for R and S for the color value of a single pixel. To address this, researchers in intrinsic decomposition have proposed numerous assumptions and prior constraints to ensure that eigendecomposition can yield a reasonable solution. Traditional eigendecomposition methods primarily transform this problem into an optimization problem with numerous prior assumptions and constraints. This limits their applicability and makes manual selection of appropriate parameters difficult. With the advancement of deep learning technology and the availability of numerous intrinsic image decomposition datasets, deep learning-based methods can extract correlations between large amounts of eigendecomposition data through network training, thereby eliminating or reducing the need for manual prior assumptions and achieving better results.

[0003] Intrinsic image decomposition technology has also been extended beyond images to video. Rather than simply performing eigendecomposition on each frame, it must ensure consistency across frames. This approach can be broadly categorized into two types. One involves performing eigendecomposition on the image first, leveraging motion information like optical flow to establish relationships between frames, and then adjusting the decomposition results for each frame through post-processing to maintain consistency across frames. The other approach involves constructing consistency constraints and incorporating relevant loss functions into the optimization process.

[0004] Neural rendering technology (NeRF) has been a hot research topic in the past two years. The NeRF network input consists of a three-dimensional spatial position (x, y, z) and a two-dimensional view direction (θ, φ). A multi-layer perceptron (MLP) is used to output the color RGB value and volume density of the point at that position and view direction. During training, a set of scene images with known camera poses is input. Rays are generated by randomly sampling pixels in the image. The points along each ray are then integrated to obtain the color of each pixel using volume rendering. Numerous works have been proposed that combine NeRF with inverse rendering. The goal is to use NeRF, a differentiable volume rendering pipeline, to simultaneously recover the surface geometry, material, and ambient lighting of objects from multiple frames, thereby enabling effects such as material editing and relighting. While these works have demonstrated some very realistic editing results, they share some common challenges. These methods rely on accurate geometry recovery. If the surface-based NeRF method cannot correctly model the geometry, accurate lighting and material restoration cannot be achieved. At the same time, since this type of method simplifies the rendering process and lighting representation, there are limitations on application scenarios. Currently, this type of inverse rendering NeRF method can only be used for object-level data.

[0005] Problems with existing methods include:

[0006] 1. Restrictions on application scenarios

[0007] Inverse rendering methods, which combine neural rendering, attempt to recover basic scene parameters such as lighting, geometry, and materials from images, thereby digitizing the scene. However, this problem is extremely difficult because a significant amount of information about the scene's lighting, geometry, and materials is lost during the image generation process, making it a highly ill-posed problem. Therefore, inverse rendering methods typically require a series of prior assumptions about scene geometry, lighting, and the rendering pipeline, as well as a relatively accurate 3D model. Consequently, these methods are typically limited to the object level.

[0008] 2. It is difficult to accurately edit the color of objects in the neural rendering framework

[0009] While the combination of inverse rendering and neural rendering can produce realistic editing effects by modifying the scene's lighting and material parameters, information such as the object's reflectivity is still implicitly represented through the MLP, making it difficult to accurately edit the object's reflectivity in this implicit representation.

[0010] 3. Perform intrinsic decomposition on the video

[0011] Performing intrinsic decomposition on a video also requires ensuring consistency across frames. Existing video intrinsic decomposition methods establish data associations between frames and incorporate spatiotemporal constraints into the optimization process to ensure consistency. However, if neural rendering methods can be used to map these decomposition attributes into the same latent space, this consistency can be naturally maintained without the need for constructing constraints. Summary of the Invention

[0012] In response to the shortcomings of the existing technology, the present invention proposes an editable novel view synthesis method based on intrinsic neural radiance field IntrinsicNeRF, which effectively overcomes the above problems and obtains good experimental results on object-level and scene-level test data.

[0013] The technical solutions of the present invention are as follows:

[0014] The present invention proposes a novel editable view synthesis method based on intrinsic neural radiation field, which includes the following steps:

[0015] S1: Construct intrinsic neural radiation field

[0016] The intrinsic neural radiation field is represented by a multi-layer perceptron (MLP) network.

[0017] S2: Unsupervised prior guidance

[0018] Introducing unsupervised prior information into the multi-layer perceptron (MLP) network, the unsupervised prior information includes chromaticity prior, reflectivity sparsity prior, illumination smoothness prior, residual constraint, and brightness prior;

[0019] S3: Network Optimization

[0020] Based on the data scale, for object-level data, we select the adaptive iterative reflectivity clustering method applicable to the object level, and for scene-level data, we select the hierarchical reflectivity clustering method applicable to the scene level. Through clustering, we obtain a predicted sparse reflectivity result, and construct the reflectivity clustering loss based on the predicted result and the actual sparse reflectivity result.

[0021] The multi-layer perceptron (MLP) network is optimized by jointly optimizing color loss, semantic loss, intrinsic decomposition unsupervised prior-related loss, and reflectance clustering loss. Once the network is trained, given a 3D space point and camera orientation, the MLP network can output the reflectance, shadow, and residual term of the point, and ultimately synthesize a novel view.

[0022] According to a preferred embodiment of the present invention, the MLP network in S1 takes the spatial point coordinates and the viewing direction as input, and outputs the volume density, the semantic label of the point, the reflectivity value R, the illumination S, and a residual term Re related to the viewing angle; wherein R, S, and Re are the intrinsic decomposition results of the point; the RGB color value of the point can be calculated through the intrinsic decomposition model I=RxS+Re, and the image under the new viewing angle can be constructed through volume rendering.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1) This invention combines intrinsic image decomposition technology with neural rendering technology. Based on the neural rendering method, it can achieve intrinsic decomposition of the image to obtain reflectance, illumination, and residual terms. Compared with other intrinsic decomposition methods, this method takes advantage of the characteristics of neural rendering and ensures the consistency of intrinsic decomposition results between different frames.

[0025] 2) This invention has no limitations on the scale of its use cases. Compared to existing methods that combine inverse rendering with neural rendering, which can only operate on object-level data, the IntrinsicNeRF proposed in this invention can operate well on both object-level and scene-level data.

[0026] 3) The present invention can edit videos in real time. Based on the information obtained by decomposing video data using IntrinsicNeRF proposed by the present invention, the present invention can perform real-time editing operations on videos, achieving effects including object color editing, light brightness adjustment, and material editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a framework diagram of the intrinsic neural radiation field IntrinsicNeRF of the present invention.

[0028] Figure 2 This is a schematic diagram of the network structure of the intrinsic neural radiation field IntrinsicNeRF proposed in the present invention.

[0029] Figure 3 It is a schematic diagram of the distance perception point sampling method in the present invention.

[0030] Figure 4 It is a schematic diagram of the process of reflectivity adaptive iterative clustering in the present invention.

[0031] Figure 5 It is a schematic diagram of the hierarchical reflectivity clustering and indexing method in the present invention.

[0032] Figure 6 This is a comparison of the reflectivity results of the method of the present invention and other similar methods on the Replica scene dataset in the present invention.

[0033] Figure 7 This is the video color editing effect based on IntrinsicNeRF in the present invention.

[0034] Figure 8 This is the light brightness editing effect based on IntrinsicNeRF in the present invention.

[0035] Figure 9 This is the material editing effect based on IntrinsicNeRF in the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0037] The present invention provides an editable novel view synthesis method based on the intrinsic neural radiation field, which realizes the temporal and spatial consistent intrinsic image decomposition of the video sequence based on the neural rendering framework, and can achieve real-time editing effects on the video, including scene object color editing, light brightness editing, and material editing effects, and can also synthesize pictures with new perspectives.

[0038] like Figure 1 As shown, the method of the present invention mainly comprises the following steps:

[0039] Step 1: Intrinsic Neural Radiance Field

[0040] A multi-layer perceptron (MLP) is constructed to represent the intrinsic neural radiance field. It takes spatial point coordinates and viewing direction as input and outputs volume density, a semantic label for that point (not available at the object level), reflectivity R, illumination S, and a view-dependent residual Re. R, S, and Re are the results of the intrinsic decomposition of the point. The RGB color value of the point can be calculated using I = RxS + Re, and the image at the new viewpoint is constructed through volume rendering. Each sampled pixel is associated with a distance to facilitate subsequent optimization of the unsupervised intrinsic prior.

[0041] Step 2: Unsupervised prior bootstrapping.

[0042] Intrinsic image decomposition is a highly ill-posed problem, as countless combinations of reflectance and illumination can produce the same RGB value. To ensure that the reflectance R, illumination S, and residual term Re output by the network can form the final RGB image while also conforming to the desired decomposition result, prior constraints commonly used in traditional intrinsic decomposition methods are introduced into the optimization process, including the monochromatic light assumption, a sparse reflectance prior, and a smooth illumination prior. This network does not require intrinsic decomposition supervision data, but instead performs unsupervised optimization based on a series of prior information.

[0043] Step 3: Network Optimization

[0044] Based on the data scale, for object-level data, we select adaptive iterative reflectance clustering for object-level data, and for scene-level data, we select hierarchical reflectance clustering for scene-level data. We obtain a predicted sparse reflectance result through clustering, and construct a reflectance clustering loss based on the predicted result and the actual sparse reflectance result. We jointly optimize the color loss, semantic loss, loss related to the unsupervised prior of intrinsic decomposition, and reflectance clustering loss. The complete loss function is as follows:

[0045] L final =λ pho L pho +λ sem L sem +λ chrom L chrom +λ reflect L reflect +λ non-local L non-local +λ shading L shading +λ cluster L cluster +λ residual L residual +λ intensity L intensity (twenty one)

[0046] In this embodiment, the image color loss weight λ pho Set to 1, the semantic loss weight λ sem is 0.04, the chroma loss weight λ chrom is 0.04, the residual loss weight λ residualIt is 1 in the early stage of training (the first 100,000 iterations) and is reduced to 0.02 in the later stage (the last 100,000 iterations). The weight of the illumination smoothness loss is set to 1, the weight of the local reflectivity loss is 0.04, and the weight of the non-local reflectivity loss is 0.02. The weight of the clustering constraint is initially set to 0.01, and then gradually increases exponentially to 1 as the iteration proceeds. The present invention uses PyTorch to implement IntrinsicNeRF and trains on a single Nvidia RTX3090 24G graphics card. Each Batch contains 1024 rays. The image resolution of the Replica scene is 320x240, and the image size of the Blender object dataset is 400x400. The Adam optimizer is used, the learning rate is set to 5e-4, and 200,000 iterations.

[0047] In the network optimization of step 3, under the guidance of unsupervised priors, the intrinsic decomposition results output by the network are very close to the target. However, in some areas with the same material and the same reflectivity, there are still inconsistent decomposed reflectivities. Therefore, the present invention proposes an adaptive iterative clustering of reflectivity applicable to the object level for object-level data: a clustering operation G is constructed that is continuously updated during the training process. For a group of similar reflectivity colors r, the operation G maps these colors to the same target reflectivity value G(r). A clustering constraint is added during the network optimization process so that the similar reflectivity values output by the network gradually converge, making the reflectivity values in the final scene more sparse and obtaining better intrinsic decomposition results.

[0048] In the network optimization of step 3, for scene-level data, due to the relatively rich number of object types in the scene, similar reflectivity values may appear on different objects. During the iterative clustering process, these similar reflectivity values on different objects will be mistakenly clustered into the same category. Therefore, the present invention further considers the semantic information of objects in the scene, restricts the prior constraints on reflectivity and the iterative clustering operation to objects of the same type, and further proposes a method for hierarchical clustering of reflectivity applicable to the scene level. The reflectivity values are stored in the form of a two-level index of (object category, reflectivity category). This storage format is also further used in the video color editing application of the present invention.

[0049] In a specific implementation of the present invention, the implementation process of step 1 is introduced.

[0050] The MLP network structure is shown in the attached Figure 2 The overall structure of the network is similar to that in NeRF. The network is based on three-dimensional coordinates (x, y, z) and viewing direction. It takes as input, and outputs volume density, semantic label (optional), reflectivity R, illumination S, and view-dependent residual term Re through multiple branches.

[0051] Different from the I=RxS model commonly used in intrinsic image decomposition, which decomposes the image into the product of reflectance R and illumination S, the present invention further uses a residual term Re to represent phenomena in the image that cannot be represented by the Lambertian surface assumption, such as metal materials, highlights on the surface of objects, etc. Therefore, the intrinsic decomposition model used in the present invention is I=RxS+Re. According to the NeRF method, by integrating R, S, and Re of each sampling point on a ray, the intrinsic decomposition result of a pixel point can be obtained, and then the color value of the pixel point can be obtained according to the formula I=RxS+Re. Construct the same pixel value loss function L as in NeRF pho :

[0052]

[0053] In a specific implementation of the present invention, the implementation process of step 2 is introduced.

[0054] If the intrinsic image decomposition is introduced into the NeRF framework without any constraints, unstable eigendecomposition results will be obtained. Therefore, the present invention needs to introduce a series of prior constraints to guide the eigendecomposition content. However, most of the traditional eigendecomposition constraints are built on adjacent pixel points. Unlike the simple eigendecomposition method that takes a complete image as input, each iterative optimization in the NeRF framework can only obtain some random sampling points, and there is no correlation between these sampling points. In order to apply the constraints of the intrinsic image decomposition to the NeRF framework, the present invention proposes a distance-aware point sampling method, as shown in the attached figure. Figure 3 As shown in the figure, similar to NeRF, each iteration optimizes and samples a total of 1024 points. First, 512 sampling points are randomly selected from the image. Then, a four-neighborhood point is randomly selected from these 512 points to obtain another 512 points. These two groups of points are spliced together to obtain a complete set of 1024 sampling points. Among them, the point numbered n (n<512) and the point numbered n+512 are adjacent point pairs, while the two points numbered n and n+256 have no connection and can be regarded as a pair of random point pairs. These adjacent point pairs and random point pairs will be used to construct the unsupervised prior.

[0055] The unsupervised prior guidance mentioned in step 2 mainly includes the following six prior constraints. The meaning of each prior and the corresponding loss function construction method are as follows:

[0056] Step 2.1 Monochromatic light assumption and chromaticity similarity weight:

[0057] In order to simplify the entire problem, the present invention uses the monochromatic light assumption commonly used in intrinsic image decomposition, that is, it is assumed that the illumination of each point in the environment is the same color, so the illumination S can be represented by only one grayscale value, thereby reducing the number of unknowns in the problem. Under this assumption, the chromaticity value of the reflectance is equal to the chromaticity value of the original image. The calculation method of chromaticity is c(x) = I(x) / |I(x)|, and the chromaticity similarity weight ω between two pixel values is defined here. cs (x, y), this weight will be used repeatedly in the subsequent prior constraints and is calculated as follows:

[0058]

[0059] Step 2.2 Chromaticity prior:

[0060] Under the assumption of monochromatic light, the chromaticity value of the reflectance should be as close as possible to the chromaticity value of the image. However, this invention does not directly set the chromaticity value of the reflectance to be equal to the chromaticity value of the image, because there is also a residual term that also has a certain color. Therefore, a chromaticity prior loss function is introduced to make the chromaticity value of the reflectance close to the chromaticity value of the image. The definition is as follows:

[0061]

[0062] Where x represents a pixel point, and c r represents the chromaticity value in the reflectance image, and c represents the chromaticity value of the RGB image.

[0063] Step 2.3 Reflectance sparsity prior:

[0064] Reflectivity is piecewise constant, which means that pixels that are spatially and chromatically close typically have the same reflectivity. Based on the pixel sampling method proposed in step 1.2 of the present invention, reflectivity constraints are constructed between adjacent pixels.

[0065]

[0066] In artificial environments, some distant points may have the same reflectivity, such as walls or floors that occupy a large area in the image space. Therefore, it is also necessary to ensure that non-local reflectivity constraints are constructed. Similarly, based on the sampling method in step 1.2, reflectivity constraints between random non-local pixels are constructed.

[0067]

[0068] The formula form of local and non-local reflectivity constraints is exactly the same, but the loss weight of the non-local constraint is lower than the loss weight of the local reflectivity constraint during training.

[0069] Step 2.4 Lighting Smoothness Prior:

[0070] The present invention constrains the change of the illumination component S to be smooth, that is, the S values of adjacent pixels are as close as possible. This prior is based on the assumption that the geometric changes of the surface of objects in the environment and the changes in illumination are smooth.

[0071]

[0072] The greater the chromaticity difference between two points, the stronger the lighting smoothing constraint.

[0073] Step 2.5 Residual constraint:

[0074] The present invention aims to represent image content as closely as possible using R and S, as diffuse light in the environment generally dominates. Therefore, a loss is added to the residual term to minimize Re. This also prevents extreme cases, such as when both R and S become zero and Re = 1, which would also minimize the previous loss functions. However, this is not the intended result of the present invention.

[0075]

[0076] In the early stages of network training, the weight of this constraint is set large, making RxS as close as possible to the target image I. Then, in the second half of training, the weight of this constraint is reduced. At this time, the network output R and S are basically stable. However, since RxS cannot represent content related to the view angle, the residual constraint is reduced to allow Re to represent the remaining content that cannot be represented by the Lambertian surface, such as object highlights, which also makes RxS+Re closer to the target image I.

[0077] Step 2.6 Brightness prior:

[0078] The previous constraints on reflectivity and shading only consider the relative relationship between two pixels, R and S, and do not constrain the absolute size of R or S. To prevent R and S from falling into extreme values during the optimization process and to keep the overall values of R and S within a reasonable range, this invention adds an overall brightness constraint:

[0079]

[0080] The average value of R output by the network is made as close as possible to the average value of the original image's RGB values. During the experiment, the weight of this constraint was set high in the early stages of training because the network is unstable at first. Strengthening this constraint can prevent the network from falling into extreme values. In the later stages of training, the weight of this constraint was reduced.

[0081] In a specific implementation of the present invention, the implementation process of the reflectivity adaptive iterative clustering applicable to object-level data in step 3 is introduced.

[0082] like Figure 4 As shown in Figure 3, the reflectivity adaptive iterative clustering mentioned in step 3 mainly includes the following five steps:

[0083] Step 3.1 Color space conversion:

[0084] During network training, a set of camera poses is extracted after every 10,000 iterations, generating complete images of the corresponding poses and their corresponding intrinsic decomposition components. The R component of each pixel P in this set of images is then extracted, and the RGB values of the reflectance R are transformed into three channels: luminance, chrominance r, and chrominance g. A coefficient β, less than 1, is added to the luminance channel. In this experiment, β is set to 0.5. This transformation aims to reduce the impact of brightness differences on clustering, allowing clustering to focus more on the similarity of reflectance and chrominance between two points. The space after the RGB transformation is called f-space.

[0085] Step 3.2 Mean-Shift Clustering:

[0086] Next, mean shift clustering is performed on the reflectance values of these pixels in f space. Unlike the K-Means method, which requires specifying the number of reflectance categories K in the scene, the mean shift clustering method can automatically determine the number of clustering categories because the present invention does not know how many categories of reflectance values there are in the scene.

[0087] Step 3.3 Clustering operation G

[0088] After the clustering step above, a set of cluster centers is obtained, along with a classification label for each pixel P. The present invention defines the clustering operation G as follows: for any reflectance value r, the category to which the pixel P closest to the reflectance value in the f-space belongs is taken as the category of the point, and the center color of the category is taken as the cluster color G(r) of the reflectance of the point.

[0089] Step 3.4 Voxel filtering

[0090] The number of pixels P is very large, and directly finding the nearest neighbors among them is very time-consuming. Because reflectance values are guided to cluster in very small areas during training, it is not necessary to find the nearest neighbors among all pixels P. Here, all pixels P are voxel-filtered in f space, and the remaining points are used as anchors. This way, operation G only needs to find the nearest neighbors among the anchors and output the center color of the category to which the anchors belong.

[0091] Step 3.5 Clustering loss function and optimization

[0092] Construct the loss between the reflectivity value r and its cluster reflectivity value G(r), and guide similar reflectivity values to move towards the same reflectivity value. The following cluster loss function is obtained, where r cluster =G(r):

[0093]

[0094] During implementation, the weight of the clustering loss will gradually increase with the number of iterations. Simultaneously, the bandwidth, a key parameter in the mean-shift algorithm, will also gradually increase (a larger bandwidth means fewer mean-shift clustering categories). Because the reflectivity R generated by the network is unreliable in the early stages of network optimization, a lower weight is required. However, using a larger weight at the end of optimization will steer the final network output towards a clustering effect, making the reflectivity output by the network essentially indistinguishable from the reflectivity after clustering.

[0095] In a specific implementation of the present invention, the implementation process of the reflectivity hierarchical clustering and indexing method applicable to scene-level data in step 3 is introduced. It mainly includes the following two steps, specifically:

[0096] Step 3.6 IntrinsicNeRF semantic branch

[0097] In scene-level data, if all pixels are clustered directly, it is very likely that similar reflectivities on different objects will be clustered into the same category, but the reflectivities on objects of different categories should not be related. For scene-level data, the present invention is based on the work of Semantic-NeRF and combines the semantic information in the scene to further propose a hierarchical clustering method. The present invention adds a branch that is only related to the input three-dimensional coordinates on the basis of IntrinsicNeRF, and outputs the semantic label of the spatial point. Similarly, the semantic information of all sampling points on a ray is integrated to obtain the semantic label of the pixel point. The Replica scene data contains the semantic label data corresponding to each frame of the image, supervises the semantic branch, and constructs the following semantic loss function L sem :

[0098]

[0099] where p I Indicates the probability value of the semantic label true value corresponding to category l, and They respectively represent the rough and fine estimated probability values of light r for category l (the values estimated by the coarse and fine networks).

[0100] Step 3.7 Semantic-based reflectivity hierarchical clustering

[0101] Different from the object-level reflectivity iterative clustering method that only needs to obtain the reflectivity value of each pixel, here we need to extract the semantic segmentation label and reflectivity value of each pixel in this set of images at the same time. According to the different semantic labels of all pixels, all pixel sets P are divided into {P1P2, ..., P N}These N subsets, where N is the number of semantic labels in the scene. Then, N clustering operations {G1G2, ..., G N} (defined in step 3.3 of the adaptive iterative clustering of reflectivity for object-level data). The present invention then defines a hierarchical clustering operation HG, which takes as input the reflectivity of a pixel and the semantic label of the pixel, and then uses the corresponding clustering operation based on the semantic label to obtain the clustered reflectivity.

[0102] HG(RGB, label x )=G x (RGB).

[0103] Where x represents the semantic category corresponding to the pixel. Such a hierarchical clustering strategy allows clustering information to be stored in a tree structure, which can be quickly queried, such as Figure 5 For each pixel, the corresponding clustering operation G is first selected based on the semantic information of the pixel. Then, the nearest anchor point in the category is found, and the center color of the category to which the anchor point belongs is used as the cluster color for the reflectivity of the pixel. This hierarchical clustering method is also further used in the video editing method of the present invention.

[0104] In order to further demonstrate the implementation effect of the present invention, this embodiment conducted experiments from four aspects:

[0105] ① First, the present invention qualitatively compares the reflectivity decomposition results and quantitatively compares the new perspective image synthesis effects with similar methods on the Replica scene dataset. ② An ablation experiment is conducted on the IntrinsicNeRF proposed in the present invention to prove the effectiveness of the traditional intrinsic decomposition prior and iterative clustering method used in the present invention. ③ Then, the present invention tests the real-time video editing effect based on the IntrinsicNeRF decomposition results, including color editing, lighting changes, and material editing.

[0106] List of datasets used:

[0107] The present invention uses 8 Blender object model data sets, 4 of which are from Invrender and 4 from NeRF. The Blender Cycles renderer is used to obtain 100 pictures of each object at different perspectives as training data, and another 200 pictures at different perspectives as test data. At the same time, the corresponding reflectivity map at each perspective is generated for quantitative evaluation of the intrinsic decomposition results. The Replica scene data set generated in the Semantic-NeRF work is also used. The data set contains video sequences recorded in 8 different rooms, corresponding depth maps, semantic segmentation labels, and camera poses for each frame. The video sequence in each scene contains 900 320x240 resolution images. An image is taken every 5 frames as training data, and the middle frame of the training data is taken as test data.

[0108] Comparison with similar methods: This paper selects a variety of similar methods for comparison with the IntrinsicNeRF proposed in this invention, including the traditional intrinsic image decomposition method IIW based on optimization, the supervised intrinsic image decomposition method CGIntrinsic based on deep learning, the unsupervised intrinsic image decomposition method USI3D based on deep learning, and the NeRF method InvRender based on inverse rendering.

[0109] Evaluation indicators:

[0110] Reflectivity evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learning-perceptual image patch similarity (LPIPS), mean square error (MSE), and local mean square error (LMSE).

[0111] New perspective synthetic image quality evaluation indicators: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Learning Perceptual Image Patch Similarity (LPIPS)

[0112] Semantic segmentation evaluation metric: mean intersection over union (mIoU)

[0113] Experiment 1: Comparison of results on the Replica scenario dataset

[0114] Comparison of reflectivity decomposition results: Since the Replica scene dataset does not provide the true reflectivity value, it is impossible to quantitatively compare the performance of different methods on this dataset. Therefore, this paper only compares the visual effects of the reflectivity decomposition results on this dataset. Figure 6 shown.

[0115] Comparison of New Perspective Image Synthesis Quality: In experiments on the Replica scene dataset, we compared the output of IntrinsicNeRF and Semantic-NeRF. In addition to comparing the quality of new perspective synthesis images, we also compared the semantic segmentation performance, as shown in Table 1. The results show that the two methods achieve very similar values across all aspects, demonstrating that the IntrinsicNeRF extension method does not significantly impact the original NeRF method and can also enhance the functionality of tasks like NeRF.

[0116] Table 1 Comparison between Semantic-NeRF and the proposed method on Replica scene data

[0117] method PSNR↑ SSIM↑ LPIPS↓ mIoU↑ Semantic-NeRF 30.9770 0.8955 0.1066 0.9725 Ours 30.7044 0.8908 0.1140 0.9702

[0118] Experiment 2: Ablation experiment

[0119] In order to verify the effectiveness of several key methods in the present invention, a series of ablation experiments are conducted here. First, on the Blender dataset, ① the baseline of the experiment is to simply transform the network structure into the IntrinsicNeRF structure based on NeRF, but no prior constraints are introduced during training and no clustering is performed. ② At the same time, the present invention tests only adding unsupervised constraints on the basis of the baseline, but without iterative clustering. ③ Finally, the complete IntrinsicNeRF method. The reflectivity estimation results are shown in Tables 2 and 3. It can be clearly seen from the quantitative comparison results in the table and the visual effects in the figure that the best effect can be achieved by fully using several key methods in this article.

[0120] Table 2 Ablation experiment results on Blender object dataset (invrender four models)

[0121]

[0122] Table 3 Ablation experiment results on Blender object dataset (four NeRF models)

[0123]

[0124] On the Replica dataset, although numerical comparison is not possible, visual comparison is possible. In the ablation experiment on the Replica dataset, in addition to the three methods used in Blender, the effect of not performing hierarchical clustering is also tested. Figure 6 As shown in the figure, if we simply cluster all reflectances instead of performing hierarchical clustering, the reflectance decomposition effect is very poor, and similar reflectances of different objects are incorrectly clustered into one category. However, the complete method that introduces hierarchical clustering significantly improves the effect.

[0125] Experiment 3: Real-time video editing effects

[0126] Scene object color editing: The reflectivity generated by the IntrinsicNeRF network will be saved in the form of [semantic category, reflectivity category]. At the same time, the reflectivity of all semantic categories corresponding to the entire scene is also saved (obtained through the last round of hierarchical iterative clustering). Therefore, the reflectivity value corresponding to each pixel can be quickly found through the [semantic category, reflectivity category] label. Based on this representation method, the color of objects in the video can be edited in real time. By simply modifying the color of a certain reflectivity category, the reflectivity values of all pixels belonging to that category in the video can be modified at the same time. Then, the modified reflectivity R, the unmodified illumination S, and the residual Re can be used to reconstruct the edited video, such as Figure 7 shown.

[0127] Light intensity editing: IntrinsicNeRF also decomposes a residual term Re. Under the influence of various constraints, the residual term Re includes parts that cannot be described by the Lambertian surface assumption, such as the highlights on the object surface and the light spots on the wall. By multiplying the Re component by a coefficient k to scale it, and then replacing the transformed Re with the original Re, we reconstruct the image I'=Rx S+kx Re, thereby achieving the effect of brightening the overall lighting in the environment. The light intensity editing test results on Blender and Replica datasets are shown below. Figure 8 .

[0128] Material editing: By performing some transformations on the intrinsic image decomposition results, illumination S and residual Re, you can simulate the effect of modifying the object's material. For example, by performing nonlinear mapping on the entire illumination S layer, making the high-brightness value part brighter and the low-brightness value part darker, you can make the object more metallic. Figure 9 shown.

[0129] Based on the above experimental results, the following conclusions can be drawn:

[0130] 1) This paper proposes IntrinsicNeRF, the first method to combine intrinsic image decomposition with neural rendering. This method not only synthesizes images from new viewpoints but also generates the image's intrinsic decomposition results (reflectance, illumination, and residual terms). This method leverages the unique characteristics of neural rendering to ensure consistency across different viewpoints.

[0131] 2) This paper proposes a point sampling method that can introduce the a priori constraints of traditional eigendecomposition methods into the optimization process of the neural rendering framework. It also proposes an iterative clustering optimization and a semantic information-based hierarchical clustering method to obtain better eigendecomposition results.

[0132] 3) The IntrinsicNeRF proposed in this paper is not limited by the scene scale and can be applied to both object-level data and scene-level data. It has achieved good experimental results on both the Blender object dataset and the Replica scene dataset.

[0133] 4) The present invention supports multiple real-time video editing functions. Based on the scene intrinsic decomposition data, it can realize real-time object color editing, material editing, and light brightness adjustment functions in video sequences, while also supporting the generation of high-quality new perspective images.

[0134] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the scope of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A novel editable view synthesis method based on intrinsic neural radiation field, characterized in that: The steps include: S1: Construct intrinsic neural radiation field The intrinsic neural radiation field is represented by a multi-layer perceptron (MLP) network. The MLP network in S1 takes spatial point coordinates and viewing direction as input, and outputs volume density, the semantic label of the point, reflectivity value R, illumination S, and a residual term Re related to the viewing angle. R, S, and Re are the intrinsic decomposition results of the point. The RGB color value of the point can be calculated through the intrinsic decomposition model I = R × S + Re, and an image under a new viewing angle can be constructed through volume rendering. S2: Unsupervised prior guidance Introducing unsupervised prior information into the multi-layer perceptron (MLP) network, the unsupervised prior information includes chromaticity prior, reflectivity sparsity prior, illumination smoothness prior, residual constraint, and brightness prior; S3: Network Optimization According to the data scale, for object-level data, we select the adaptive iterative reflectivity clustering method applicable to the object level, and for scene-level data, we select the hierarchical reflectivity clustering method applicable to the scene level. A predicted sparse reflectance result is obtained through clustering, and a reflectance clustering loss is constructed based on the predicted result and the actual sparse reflectance result; The adaptive iterative clustering of reflectivity applicable to the object level is as follows: for the object level data, a clustering operation G is constructed which is continuously updated during the training process; For a set of similar reflectance colors r, the operation G will map these colors to the same target reflectance value G(r). During the network optimization process, the similar reflectance values output by the network are gradually brought closer together, making the reflectance values in the final scene more sparse and obtaining better intrinsic decomposition results. The multi-layer perceptron (MLP) network is optimized by jointly optimizing color loss, semantic loss, intrinsic decomposition unsupervised prior-related loss, and reflectance clustering loss. Once the network is trained, given a 3D space point and camera direction, the MLP network outputs the reflectance, shadow, and residual term of the point, and ultimately synthesizes a novel view.

2. The novel editable view synthesis method based on intrinsic neural radiation field according to claim 1 is characterized in that: The unsupervised prior guidance described in S2 specifically includes the following steps: Step 2.1 Monochromatic light assumption and chromaticity similarity weight: Assume that the illumination at each point in the environment is the same color, and the illumination S is represented by a grayscale value; the chromaticity value of the reflectance is equal to the chromaticity value of the original image; the chromaticity is calculated as c(x) = I(x) / |I(x)|, where I(x) represents the pixel color of the image, and the chromaticity similarity weight ω between two pixel values is defined. cs (x, y), calculated as follows: Among them, α cs =60,|| || 2 represents the two-norm; Step 2.2 Chromaticity prior: A chromaticity prior loss function is introduced to make the reflectance chromaticity value close to the image chromaticity value, which is defined as follows: Where x represents a pixel point, and c r represents the chromaticity value in the reflectance image, and c represents the chromaticity value of the RGB image; Step 2.3 Reflectance sparsity prior: Construct reflectivity constraints between local adjacent pixels: Here r(x) represents the reflectivity value at coordinate x; Represents the adjacent area of coordinate x; Construct reflectivity constraints between random non-local pixels: Here r(x) represents the reflectivity value at coordinate x; Represents the local non-adjacent region of coordinate x; Among them, the loss weight of the non-local constraint is lower than the loss weight of the local reflectivity constraint during training; Step 2.4 Lighting Smoothness Prior: The change of the illumination component S is constrained to be smooth, that is, the S values of adjacent pixels are as close as possible; this prior is based on the assumption that the geometric changes and illumination changes of the object surface in the environment are smooth. Where s(x) represents the value of the s component at coordinate x. The larger the chromaticity difference between two points, the stronger the illumination smoothing constraint. Step 2.5 Residual constraint: Set up and add residual constraints: In the early stage of network training, the weight of the residual constraint is set large to make R×S as close to the target image I as possible, and the weight of the residual constraint is reduced in the second half of network training; Step 2.6 Brightness prior: Add an overall brightness constraint: The average R value of the network output is made as close as possible to the average value of the original image RGB value. The weight of the overall brightness constraint is set larger in the early stage of training, and reduced in the later stage of training.

3. The novel editable view synthesis method based on intrinsic neural radiation field according to claim 2, characterized in that: When constructing the reflectance sparsity prior, the pixels are sampled using the distance-aware point sampling method. The specific distance-aware point sampling method is as follows: a total of 1024 points are sampled in each iteration; first, 512 sampling points are randomly selected in the image, and then a four-neighborhood point is randomly selected from these 512 points to obtain another 512 points; these two groups of points are spliced together to obtain a complete 1024 sampling points; among them, when the number n is less than 512, the point numbered n and the point numbered n+512 are adjacent point pairs, while the two points numbered n and n+256 have no association, so they can be regarded as a pair of random point pairs.

4. The novel editable view synthesis method based on intrinsic neural radiation field according to claim 1, characterized in that: The object-level reflectivity adaptive iterative clustering method specifically includes the following steps: Step 3.1 Color space conversion: During network training, a set of camera poses is extracted after each set number of iterations, and images of the corresponding poses and the corresponding intrinsic decomposition components are completely generated. The R components of all pixels P contained in this set of images are extracted, and the RGB values of the reflectivity R are transformed into three channels: brightness, chroma r, and chroma g. A coefficient β less than 1 is added before the brightness channel to reduce the impact of brightness differences on clustering. The space after RGB transformation is called f space. Step 3.2 Mean-Shift Clustering: Perform mean shift clustering on the reflectance values of these pixels in the f space; Step 3.3 Clustering operation G After clustering in step 3.2, a set of cluster centers is obtained, and a classification label for each pixel P is also obtained. The clustering operation G is defined as follows: for any reflectance value r, the category of the pixel P to which the reflectance is mapped closest in the f space is used as the category of the point, and the center color of the category is used as the cluster color G(r) of the reflectance of the point. Step 3.4 Voxel filtering Perform voxel filtering on all pixels P in the f space, and use the remaining points as anchors. In this way, the operation G only needs to find the nearest neighbors in the anchors and then output the center color of the category to which the anchors belong. Step 3.5 Clustering loss function and optimization Construct the loss between the reflectivity value r and its cluster reflectivity value G(r), guiding similar reflectivity values to move towards the same reflectivity value; the following cluster loss function is obtained, where r cluster =G(r):

5. The novel editable view synthesis method based on intrinsic neural radiation field according to claim 1, characterized in that: The hierarchical clustering of reflectivity applicable to the scene level is: 1) By adding a semantic branch to Intrinsic NeRF, each point not only obtains reflectance, shadow, and residual terms, but also obtains a semantic label for each point; 2) Using the semantic label and reflectance value of each point, these points are first classified according to the semantic label to obtain 3D points in different semantic categories; then, for the 3D points in each semantic category, reflectance adaptive iterative clustering applicable to the object level is performed to perform iterative clustering optimization.

6. The novel editable view synthesis method based on intrinsic neural radiation field according to claim 1 or 5, characterized in that: The hierarchical clustering of reflectivity applicable to the scene level is: First, a branch related only to the input 3D coordinates is added to Intrinsic-NeRF to output the semantic label of the spatial point. In this way, each 3D point has semantic information in addition to the component of the intrinsic radiation field. Similarly, the semantic information of all sampling points on a ray is integrated to obtain the semantic label of the pixel. The scene data contains the semantic label data corresponding to each frame of the image, which supervises the semantic branch and constructs the following semantic loss function L sem : Where p′ represents the probability value of the semantic label true value corresponding to category l, and They represent the rough and fine estimated probability values of light r for category l respectively; Secondly, according to the different semantic labels of all pixels, all pixel sets P are divided into N subsets {P1P2, ..., P N }, where N is the number of semantic labels in the scene; then N clustering operations {G1G2, ..., G N Each clustering operation is defined as follows: for the reflectance value r of a pixel point in its corresponding subset, the category to which the pixel P closest to the reflectance is mapped in space belongs is used as the category of the point, and the center color of the category is used as the cluster color G(r) of the reflectance of the point; Then, a hierarchical clustering operation HG is defined, with the input being the reflectance of a pixel and the semantic label of the pixel. Then, the corresponding clustering operation is used according to the semantic label to obtain the reflectance after clustering. HG(RGB,label x )=G x (RGB) Where x represents the semantic category corresponding to the pixel; such a hierarchical clustering strategy allows clustering information to be stored in a tree structure and can be quickly queried; that is, for each pixel point, the corresponding clustering operation G is selected based on the semantic information of the point, and then the nearest anchor point in the category is found, and the center color of the category to which the anchor point belongs is used as the cluster color of the reflectance of the pixel point.

Images