Indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy

Abstract

The application relates to an indoor scene reconstruction method based on a pre-sampling and progressive semantic modeling strategy; comprising the following steps: constructing a scene reconstruction model, wherein the scene reconstruction model comprises a DFormer module, a main model module and a semantic modeling module; training the constructed scene reconstruction model based on an NYUv2 dataset; inputting an RGB image to be reconstructed and a corresponding depth map into the trained scene reconstruction model; the main model module performs scene rendering pre-sampling on the input RGB image and the corresponding depth map, obtains a rough volume density estimation, and performs formal volume and color rendering; the DFormer module converts the RGB image into a semantic image; the semantic modeling module takes the semantic image and the corresponding depth map as input, and assists in refining the volume and color rendering effect through a progressive semantic modeling strategy; and the indoor scene reconstruction is realized. The application can significantly improve the efficiency and precision of scene reconstruction, realize more accurate detail restoration and a more efficient rendering process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image processing, and more particularly, to an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy. BACKGROUND

[0002] Indoor scene reconstruction is a technology for generating three-dimensional models from multi-view image data, widely used in virtual reality, augmented reality, architectural design and other fields. This technology mainly focuses on the spatial structure and detail information of the scene, by capturing image information from different perspectives, using computer vision and image processing methods to reconstruct the real indoor environment. Traditional reconstruction methods usually rely on multi-view geometry (MVG) or voxel-based three-dimensional modeling techniques. Although these methods can provide high accuracy under certain conditions, they have high computational complexity, large storage requirements, and high quality requirements for input data. Especially in handling large-scale and complex indoor environments, it is easy to have slow rendering speed, loss of details and other problems. In addition, since traditional methods mostly use explicit representations such as voxel grids, such methods often struggle to capture subtle structures and texture changes. In recent years, implicit neural representations (such as NeRF) have gradually become an effective three-dimensional reconstruction method, by implicitly storing scene information in a multi-layer perceptron (MLP), achieving efficient modeling and rendering of three-dimensional scenes. Although such methods have significantly improved in accuracy, their demand for computing resources is still high, and the rendering efficiency is often limited by the processing speed of complex scenes. In addition, such methods have limitations in scene understanding, and the information extraction effect is not good for occlusion and incomplete view conditions.

[0003] In order to improve the efficiency and accuracy of scene reconstruction, it is necessary to introduce various information to assist in the understanding and modeling of the scene. Compared with traditional scene reconstruction methods, semantic information can help the model better identify and understand the distribution of objects in the scene, thus effectively solving the problem of unclear edge information caused by blurred frames in RGB images. At the same time, the application of pre-sampling technology can quickly capture the overall structure of the scene, reducing computational complexity. Therefore, how to combine semantic information and pre-sampling technology has become the key to improving the efficiency and accuracy of scene reconstruction. SUMMARY

[0004] The purpose of the present application is to propose an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, aiming to solve the limitations of existing methods in reconstruction efficiency and accuracy. This method can quickly capture the overall structure of the scene, effectively reduce the computational complexity, and through the progressive from coarse to fine accuracy, realize the step-by-step optimization of scene rendering, improve the model's understanding ability of complex scenes, thus realizing efficient and accurate indoor scene reconstruction.

[0005] To achieve the above object, the application adopts the following technical scheme: an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, comprising the following steps:

[0006] Step S1: constructing a scene reconstruction model, which comprises a DFormer module, a main model module and a semantic modeling module;

[0007] Step S2: training the constructed scene reconstruction model based on the NYUv2 dataset;

[0008] Step S3: inputting the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model;

[0009] The main model module performs scene rendering pre-sampling on the input RGB image and corresponding depth map, obtains rough volume density estimation, and performs formal volume and color rendering to complete the main reconstruction process of the scene;

[0010] The DFormer module converts the RGB image into a semantic image;

[0011] The semantic modeling module takes the semantic image and the corresponding depth map as input, and assists in refining the rendering effect of volume and color through a progressive semantic modeling strategy;

[0012] The indoor scene is reconstructed according to the output results of the main model module and the semantic modeling module.

[0013] Preferably, the main model module comprises a sampling-guided multi-layer perception machine SG-MLP, a truncated signed distance field multi-layer perception machine SDF MLP and a color multi-layer perception machine RGB MLP; the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and corresponding depth map, and performs rough volume density estimation on the scene; the SDF MLP and the RGB MLP of the main model module both take the output of the SG-MLP as input to perform independent rendering of volume and color, respectively.

[0014] Preferably, the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and corresponding depth map, and performs rough volume density estimation on the scene, specifically, using the frequency encoding function γ of NeRF to perform high-frequency encoding on each sampling point in the sampling point set , then performing volume density estimation on each high-frequency encoded sampling point and outputting i , the calculation formula is as follows:

[0015] ρ i =MLP θ (γ(z i ))

[0016] where MLP θ denotes the SG-MLP network, γ(z i ) is the high-frequency encoding of the sampling point z i , and the spatial information is enhanced by the frequency encoding function γ in NeRF.

[0017] Preferably, the semantic modeling module comprises a truncated signed distance field multi-layer perception machine SDF MLP and a color multi-layer perception machine RGB MLP independent of the main model module; the SDF MLP and the RGB MLP of the semantic modeling module both take the semantic image and the corresponding depth map as input, and assist in refining the rendering effect of the volume and the color through the progressive semantic modeling strategy.

[0018] Preferably, the progressive semantic modeling strategy is as follows:

[0019] The total number of rendering rounds, the coarse-precision voxel size, and the fine-precision voxel size are preset; the semantic image and the corresponding depth map are taken as the input of the coarse-precision rendering stage SDF MLP and RGB MLP and the input of the fine-precision rendering stage SDF MLP and RGB MLP; first, the volume and color rendering are performed for half of the rounds using coarse precision, and the SDF value and the RGB value of each voxel under coarse precision are output by the SDF MLP and the RGB MLP respectively; then, the volume and color rendering are performed for the second half of the rounds using fine precision, and the SDF value and the RGB value of each voxel under fine precision are output by the SDF MLP and the RGB MLP respectively, wherein the ray weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel under coarse precision is used for calculating the ray weight of the coarse-precision rendering stage, and the RGB value of each voxel under coarse precision is used for calculating the RGB value of the fine-precision stage.

[0020] Preferably, the voxel size in the fine-precision rendering stage is the same as the voxel size in the main model module rendering stage, and each dimension of the voxel in the coarse-precision rendering stage is 10 times the size of the voxel in the fine-precision rendering stage.

[0021] Preferably, the ray weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel under coarse precision is used for calculating the ray weight of the coarse-precision rendering stage, and the RGB value of each voxel under coarse precision is used for calculating the RGB value of the fine-precision stage, which is as follows:

[0022] In the coarse-precision rendering stage, the calculation of the ray weight depends on the volume density ρ i of the sampling point, and the ray weight ω coarse (k) of the coarse-precision stage is calculated using the NEUS weight calculation method:

[0023]

[0024] ρ j =σ(sdf j ·inv_s)

[0025] Where, ω coarse (k) represents the ray weight of the k-th voxel in the coarse-precision rendering stage, ρ k This represents the sampling point z corresponding to the k-th voxel. k The volume density, Δz k =z k+1 -z k ρ represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage. j This represents the SDF value (sdf) of the j-th voxel under coarse precision. j The calculated volume density, inv_s is the scaling factor preset in the coarse-precision rendering stage, and σ is the Sigmoid activation function;

[0026] In the fine-precision rendering stage, ray weights are calculated based on the weights from the coarse-precision rendering stage. The ray weight ω in the fine-precision stage... fine (l) The calculation formula is as follows:

[0027]

[0028] Where, ω fine (l) represents the ray weight of the l-th voxel in the fine-precision rendering stage, ω coarse (l) represents the ray weight of the voxel in the coarse rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, ρ l This represents the sampling point z corresponding to the l-th voxel. l The volume density, Δz l =z l+1 -z l This represents the distance between the sampling points corresponding to two adjacent voxels during the fine-precision rendering stage.

[0029] RGB value c in fine-precision rendering stage fine The calculation of (l) is as follows:

[0030] c fine (l)=MLP rgb (c coarse (l))

[0031] Among them, c fine (l) represents the RGB value of the l-th voxel at fine precision, c coarse (l) represents the RGB value of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, MLP. rgb This is an RGB MLP network with fine precision.

[0032] Preferably, the total loss function in the training of the scene reconstruction model is:

[0033]

[0034] wherein, is the loss of the main model module, is the loss of the SG-MLP pair density estimation, is the loss of the semantic modeling module, are respectively the color error loss, the depth error loss and the SDF value error loss between the actual and the rendering of the main model module, is the loss for the free space outside the object, is the smoothness loss when the signed distance field is away from the surface, is the smoothness loss to encourage the continuity and smoothness of the surface geometry; are respectively the color error loss and the SDF value error loss between the actual and the rendering of the semantic modeling module; λ main , λ SG , λ sem , λ rgb , λ sdf , λ d , λ smo , λ eik , λ fs , λ' rgb , λ' sdf are the corresponding hyperparameters.

[0035] Preferably, the reconstruction of the indoor scene according to the output results of the main model module and the semantic modeling module specifically comprises:

[0036] The main model module generates a geometric representation of the scene by calculating the SDF value, the RGB value and the ray weight ω main (l) of each voxel;

[0037] The semantic modeling module determines the voxel size of the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel size of the main model module, and obtains the ray weight ω fine (l), the SDF value and the RGB value of the fine-precision stage by calculation to improve the rendering precision; the RGB value obtained by the fine-precision rendering is converted into the semantic label corresponding to each voxel through color mapping to enhance the semantic information representation of the scene;

[0038] The Marching Cubes algorithm is used to extract the grid surface from the SDF value of each voxel in the fine precision stage of the main model module and the semantic modeling module, and the main model module is combined with the rendering result of the semantic modeling module in the fine precision to generate a three-dimensional indoor scene grid containing geometric structure and texture details, thereby realizing the reconstruction of the indoor scene.

[0039] Preferably, the combining of the main model module with the rendering result of the semantic modeling module in the fine precision is specifically:

[0040]

[0041] ρ m =σ(sdf m ·inv_s′)

[0042] where I final represents the final rendering result, ω main (l) is the ray weight of the lth voxel calculated by the main model module, c l is the RGB value of the lth voxel of the main model module, L is the number of voxels, β is a hyperparameter, ρ m represents the volume density calculated by the SDF value sdf m of the mth voxel of the main model module, ρ l represents the volume density of the sampling point z l corresponding to the lth voxel of the main model module, Δz l =z l+1 -z l represents the distance between the sampling points corresponding to two adjacent voxels in the main model module, inv_s' is a scaling factor preset by the main model module, and σ is a Sigmoid activation function.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] The method proposed in the present application can significantly reduce the computational complexity through pre-sampling technology, and gradually optimize scene rendering by adopting a progressive semantic modeling strategy, thereby enhancing the understanding of the model to the scene structure. The method enables the model to more accurately capture the detail changes in the scene, gradually refine the volume and color features, and finally realize more efficient and more accurate indoor scene reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a schematic diagram of the principle of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions of the present application will be specifically described below with reference to the drawings.

[0047] It should be noted that the following detailed description is illustrative only and is intended to provide further description in connection with the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application; as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0049] The model of the present application combines the advantages of pre-sampling and progressive semantic modeling, efficiently utilizes semantic information in the rendering process, and improves the scene structure, detail understanding and reconstruction accuracy. The preliminary estimation of the volume is realized by SG-MLP, which effectively reduces the computational complexity, and at the same time of the main model rendering, the progressive semantic modeling strategy is adopted to gradually capture rich volume and color features. Compared with traditional reconstruction methods, the present application can more accurately restore scene details and realize high-precision indoor scene reconstruction.

[0050] As shown in Figure 1 The present embodiment provides an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, comprising the following steps:

[0051] Step S1: constructing a scene reconstruction model, the scene reconstruction model comprising a DFormer module, a main model module and a semantic modeling module;

[0052] Step S2: training the constructed scene reconstruction model based on the NYUv2 dataset;

[0053] Step S3: inputting the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model;

[0054] The main model module performs scene rendering pre-sampling on the input RGB image and corresponding depth map, obtains rough volume density estimation, and performs formal volume and color rendering to complete the main reconstruction process of the scene;

[0055] The DFormer module pre-processes the RGB image, and through the pre-training stage combined with the label mapping of the NYUv2 dataset, realizes the conversion of the RGB image into a semantic image;

[0056] The semantic modeling module takes the semantic image and the corresponding depth map as input, and assists in refining the rendering effect of volume and color through the progressive semantic modeling strategy, and assists the main model to realize more fine scene restoration.

[0057] Reconstruction of the indoor scene is achieved according to the output results of the main model module and the semantic modeling module.

[0058] In the embodiment, the main model module includes a sampling-guided multi-layer perception machine (SG-MLP), a truncated signed distance field multi-layer perception machine (SDF MLP) and a color multi-layer perception machine (RGB MLP); the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and the corresponding depth map, performs rough volume density estimation on the scene, and does not involve color rendering in the process; the SDF MLP and the RGB MLP of the main model module both take the output of the SG-MLP as input to perform independent rendering of volume and color respectively.

[0059] In the embodiment, the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and the corresponding depth map, and performs rough volume density estimation on the scene, specifically, using the frequency coding function γ of NeRF to perform high-frequency coding on each sampling point in the sampling point set , and then performing volume density estimation on each high-frequency coded sampling point and outputting ρ i , the calculation formula is as follows:

[0060] ρ i =MLP θ (γ(z i ))

[0061] Wherein, MLP θ represents the SG-MLP network, γ(z i ) is the high-frequency coding of the sampling point z i , and the spatial information is enhanced through the frequency coding function γ in NeRF.

[0062] In the embodiment, the semantic modeling module includes a truncated signed distance field multi-layer perception machine (SDF MLP) and a color multi-layer perception machine (RGB MLP) independent of the main model module; the SDF MLP and the RGB MLP of the semantic modeling module both take the semantic image and the corresponding depth map as input, and assist in refining the rendering effect of volume and color through a progressive semantic modeling strategy (PFSMS); first, a coarse-precision feature plane is used for fast preliminary rendering; then switch to a regular fine-precision mode, and use independent MLP to perform fine rendering of volume and color.

[0063] In the embodiment, the progressive semantic modeling strategy optimizes the calculation efficiency and reconstruction quality through step-by-step switching of coarse-precision and fine-precision, specifically as follows:

[0064] The total number of preset rendering rounds, coarse-precision voxel size and fine-precision voxel size are set; the semantic image and the corresponding depth map are respectively taken as the input of the coarse-precision rendering stage SDF MLP and RGB MLP and the input of the fine-precision rendering stage SDF MLP and RGB MLP; first, half of the rounds of volume and color rendering are performed using coarse-precision, and the SDF value and the RGB value of each voxel under coarse-precision are respectively output by the SDF MLP and the RGB MLP; then, the latter half of the rounds of volume and color rendering are performed by switching to fine-precision, and the SDF value and the RGB value of each voxel under fine-precision are respectively output by the SDF MLP and the RGB MLP, wherein the ray weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel under coarse-precision is used for calculating the ray weight of the coarse-precision rendering stage, and the RGB value of each voxel under coarse-precision is used for calculating the RGB value of the fine-precision stage.

[0065] In the embodiment, the voxel size in the fine-precision rendering stage is the same as the voxel size in the main model module rendering stage, and each dimension of the voxel in the coarse-precision rendering stage is 10 times the voxel size in the fine-precision rendering stage.

[0066] In the embodiment, the ray weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel under coarse-precision is used for calculating the ray weight of the coarse-precision rendering stage, and the RGB value of each voxel under coarse-precision is used for calculating the RGB value of the fine-precision stage, which is as follows:

[0067] In the coarse-precision rendering stage, the calculation of the ray weight depends on the volume density ρ i of the sampling point coarse (k) is calculated using the NEUS weight calculation method:

[0068]

[0069] ρ j = σ(sdf j ·inv_s)

[0070] wherein ω coarse (k) represents the ray weight of the kth voxel in the coarse-precision rendering stage, ρ k represents the volume density of the sampling point z k corresponding to the kth voxel, Δz k =z k+1 -z k represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage, and ρ j represents the SDF value sdf jThe calculated volume density, inv s is a scaling factor preset in the coarse-precision rendering stage, and sigma is a Sigmoid activation function;

[0071] In the fine-precision rendering stage, the ray weight is calculated based on the weight of the coarse-precision rendering stage, and the ray weight of the fine-precision stage is fine (l) is calculated as follows:

[0072]

[0073] Wherein, omega fine (l) represents the ray weight of the lth voxel in the fine-precision rendering stage, omega coarse (l) represents the ray weight of the voxel corresponding to the lth voxel in the coarse-precision rendering stage, rho l represents the volume density of the sampling point z l of the lth voxel, delta z l =z l+1 -z l represents the distance between the sampling points corresponding to the adjacent two voxels in the fine-precision rendering stage.

[0074] The calculation of the RGB value c fine (l) of the fine-precision rendering stage is as follows:

[0075] c fine (l) = MLP rgb (c coarse (l))

[0076] Wherein, c fine (l) is the RGB value of the lth voxel in the fine-precision, and c coarse (l) is the RGB value of the voxel corresponding to the lth voxel in the coarse-precision rendering stage, and MLP rgb is the RGB MLP network in the fine-precision.

[0077] In this embodiment, the total loss function in the training of the scene reconstruction model is:

[0078]

[0079] Wherein, is the loss of the main model module, is the loss of the SG-MLP to the density estimation, is the loss of the semantic modeling module, respectively, are the color error loss, the depth error loss and the SDF value error loss between the actual and the rendering of the main model module, is the loss for the free space outside the object, to constrain the smoothness loss of the signed distance field away from the surface, to encourage the continuity and smoothness of the surface geometry; respectively, are the color error loss and SDF value error loss between the actual and the rendering of the semantic modeling module; λ main , λ SG , λ sem , λ rgb , λ sdf , λ d , λ smo , λ eik , λ fs , λ' rgb , λ' sdf are the corresponding hyperparameters. Wherein, The calculation of each of the above is prior art, and the specific expression is as follows:

[0080]

[0081] In the embodiment, the reconstruction of the indoor scene according to the output results of the main model module and the semantic modeling module is specifically:

[0082] The main model module generates a geometric representation of the scene by calculating the SDF value, RGB value and ray weight ω main (l) of each voxel;

[0083] The semantic modeling module determines the voxel size of the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel size of the main model module, and obtains the ray weight ω fine (l), SDF value and RGB value of the fine-precision stage by calculation to improve the rendering precision; the RGB value obtained by fine-precision rendering is converted into the semantic label corresponding to each voxel through color mapping to enhance the semantic information representation of the scene;

[0084] The Marching Cubes algorithm is used to extract the grid surface from the SDF value of each voxel of the fine-precision stage of the main model module and the semantic modeling module, and the rendering results of the main model module and the semantic modeling module under fine-precision are combined to optimize the geometric shape and texture information of the scene, generate a three-dimensional indoor scene grid containing geometric structure and texture details, and realize the reconstruction of the indoor scene.

[0085] In the embodiment, the combination of the rendering results of the main model module and the semantic modeling module under fine-precision is specifically:

[0086]

[0087] ρ m = σ(sdf m ·inv_s')

[0088] where I final denotes the final rendering result, ω main (l) is the ray weight of the lth voxel calculated by the main model module, c l is the RGB value of the lth voxel of the main model module, L is the number of voxels, β is a hyperparameter, ρ m denotes the SDF value sdf m calculated by the mth voxel of the main model module, ρ l denotes the volume density of the sampling point z l corresponding to the lth voxel of the main model module, Δz l =z l+1 -z l denotes the distance between the sampling points corresponding to two adjacent voxels in the main model module, inv_s' is a scaling factor preset by the main model module, and σ is a Sigmoid activation function.

[0089] In summary, in view of the limited ability of the prior art method to capture semantic and spatial details in the scene reconstruction process, and the low efficiency and poor accuracy of the reconstruction, an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy is proposed. The model is composed of a main model and a semantic modeling module. The main model uses SG-MLP for preliminary rendering pre-sampling to obtain rough structural information of the scene, effectively reducing the computational complexity. The semantic modeling module adopts a progressive semantic modeling strategy, gradually fusing coarse and fine accuracy feature planes, quickly constructing a scene framework at coarse accuracy, and then switching to fine accuracy for fine rendering, thereby better capturing complex details and semantic features in the scene. Compared with traditional methods, the present application can significantly improve the efficiency and accuracy of scene reconstruction, achieving more accurate detail restoration and more efficient rendering process.

[0090] The above only describes the preferred embodiments of the present application, and any equivalent changes and modifications made within the scope of the present application should be included in the scope of the present application.

Claims

1. An indoor scene reconstruction method based on presampling and progressive semantic modeling strategies, characterized by: Includes the following steps: Step S1: Construct a scene reconstruction model, which includes a DFormer module, a main model module, and a semantic modeling module; Step S2: Train the constructed scene reconstruction model based on the NYUv2 dataset; Step S3: Input the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model; The main model module performs scene rendering presampling on the input RGB image and the corresponding depth map to obtain a rough volume density estimate, and then performs formal volume and color rendering to complete the main scene reconstruction process. The DFormer module converts RGB images into semantic images; The semantic modeling module takes semantic images and corresponding depth maps as input, and uses a progressive semantic modeling strategy to help refine the rendering effects of volume and color. The indoor scene is reconstructed based on the outputs of the main model module and the semantic modeling module. The SG-MLP of the main model module performs rendering presampling on the input RGB image and corresponding depth map to perform a coarse volume density estimation of the scene. Specifically, it uses the NeRF frequency coding function γ to perform sampling on the sample point set. Each sampling point in the image is subjected to high-frequency encoding, and then the volume density of each high-frequency encoded sampling point is estimated and ρ is output. i The calculation formula is as follows: ρ i =MLP θ (γ(z i )) Among them, MLP θ Represents the SG-MLP network, γ(z) i ) is the sampling point z i High-frequency coding enhances spatial information through the frequency coding function γ in NeRF; The ray weights from the coarse-precision rendering stage carry over to the fine-precision rendering stage. The SDF value of each voxel in the coarse-precision stage is used to calculate the ray weights, and the RGB value of each voxel in the coarse-precision stage is used to calculate the RGB values ​​in the fine-precision stage, as follows: In the coarse-precision rendering stage, the calculation of ray weights depends on the volume density ρ of the sampling points. i The light weight ω in the coarse-precision stage coarse (k) Using the NEUS weight calculation method: r j =σ(sdf j ·inv_s) Where, ω coarse (k) represents the ray weight of the k-th voxel in the coarse-precision rendering stage, ρ k This represents the sampling point z corresponding to the k-th voxel. k The volume density, Δz k =z k+1 -z k ρ represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage. j This represents the SDF value (sdf) of the j-th voxel under coarse precision. j The calculated volume density, inv_s is the scaling factor preset in the coarse-precision rendering stage, and σ is the Sigmoid activation function; In the fine-precision rendering stage, ray weights are calculated based on the weights from the coarse-precision rendering stage. The ray weight ω in the fine-precision stage... fine (l) The calculation formula is as follows: Where, ω fine (l) represents the ray weight of the l-th voxel in the fine-precision rendering stage, ω coarse (l) represents the ray weight of the voxel in the coarse rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, ρ l This represents the sampling point z corresponding to the l-th voxel. l The volume density, Δz l =z l+1 -z l This represents the distance between the sampling points corresponding to two adjacent voxels during the fine-precision rendering stage. RGB value c in fine-precision rendering stage fine The calculation of (l) is as follows: c fine (l)=MLP rgb (c coarse (l)) Among them, c fine (l) represents the RGB value of the l-th voxel at fine precision, c coarse (l) represents the RGB value of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, MLP. rgb This is an RGB MLP network with fine precision.

2. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 1, characterized in that, The main model module includes a sampled guided multilayer perceptron (SG-MLP), a truncated symbolic distance field multilayer perceptron (SDF MLP), and a color multilayer perceptron (RGB MLP). The SG-MLP of the main model module performs rendering presampling on the input RGB image and the corresponding depth map to perform a coarse volume density estimation of the scene. The SDF MLP and RGB MLP of the main model module each use the output of the SG-MLP as input to perform independent volume and color rendering, respectively.

3. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 1, characterized in that, The semantic modeling module includes a truncated symbolic distance field multilayer perceptron (SDFMLP) and a color multilayer perceptron (RGB MLP), which are independent of the main model module. Both the SDF MLP and RGB MLP of the semantic modeling module take semantic images and corresponding depth maps as inputs, and use progressive semantic modeling strategies to help refine the rendering effects of volume and color.

4. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 3, characterized in that, The progressive semantic modeling strategy is as follows: The system presets the total number of rendering rounds, coarse-precision voxel size, and fine-precision voxel size. Semantic images and their corresponding depth maps are used as inputs to the SDF MLP and RGB MLP in the coarse-precision rendering stage, and to the SDF MLP and RGB MLP in the fine-precision rendering stage, respectively. First, half of the rounds of volume and color rendering are performed using coarse precision, with the SDF and RGB MLP outputting the SDF and RGB values ​​of each voxel in coarse precision, respectively. Then, the system switches to fine precision for the second half of the rounds of volume and color rendering, with the SDF and RGB MLP outputting the SDF and RGB values ​​of each voxel in fine precision, respectively. The ray weights from the coarse-precision rendering stage are carried over to the fine-precision rendering stage. The SDF values ​​of each voxel in coarse precision are used to calculate the ray weights in the coarse-precision rendering stage, and the RGB values ​​of each voxel in coarse precision are used to calculate the RGB values ​​in the fine-precision stage.

5. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 4, characterized in that, In the fine-precision rendering stage, the voxel size is the same as that in the main model module rendering stage, and in the coarse-precision rendering stage, each dimension of the voxel is 10 times the size of the voxel in the fine-precision rendering stage.

6. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 4, characterized in that, The total loss function during the training of the scene reconstruction model is: in, The loss of the main model module, The loss of SG-MLP for density estimation, For the loss of the semantic modeling module, These represent the color error loss, depth error loss, and SDF value error loss between the actual and rendered versions of the main model module. The main model module addresses the loss in the free space outside the object. To constrain the smoothness loss when the symbol distance field is far from the surface, To encourage the continuity and smoothness of surface geometry, a smoothing loss is applied; These represent the color error loss and SDF value error loss between the actual and rendered semantic modeling modules, respectively. main , λ SG , λ sem , λ rgb , λ sdf , λ d , λ smo , λ eik , λ fs 、λ' rgb 、λ' sdf For the corresponding hyperparameters.

7. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 1, characterized in that, The process of reconstructing the indoor scene based on the outputs of the main model module and the semantic modeling module is as follows: The main model module calculates the SDF value, RGB value, and ray weight ω for each voxel. main (l) generates the geometric representation of the scene; The semantic modeling module determines the voxel sizes for the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel sizes of the main model module, and calculates the ray weight ω for the fine-precision stage. fine (l) SDF and RGB values ​​are used to improve rendering accuracy; The RGB values ​​obtained from fine-precision rendering are converted into semantic labels corresponding to each voxel through color mapping to enhance the semantic information representation of the scene; The Marching Cubes algorithm is used to extract the mesh surface from the SDF value of each voxel in the fine-precision stage of the main model module and the semantic modeling module. The rendering results of the main model module and the semantic modeling module in the fine-precision stage are combined to generate a 3D indoor scene mesh containing geometric structure and texture details, thereby realizing the reconstruction of the indoor scene.

8. The indoor scene reconstruction method based on presampling and progressive semantic modeling strategy according to claim 7, characterized in that, The combination of the fine-precision rendering results of the main model module and the semantic modeling module is specifically as follows: r m =σ(sdf m ·inv_s') Among them I final ω represents the final rendering result. main (l) is the ray weight of the l-th voxel calculated by the main model module, c l This is the RGB value of the l-th voxel in the main model module, where L is the number of voxels, β is the hyperparameter, and ρ is the value of the voxel. m This represents the SDF value (sdf) of the m-th voxel in the main model module. m Calculated bulk density, ρ l This represents the sampling point z corresponding to the l-th voxel in the main model module. l The volume density, Δz l =z l+1 -z l Inv_s represents the distance between sampling points corresponding to two adjacent voxels in the main model module, σ is the scaling factor pre-set by the main model module, and σ is the Sigmoid activation function.

Images