Indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy

Abstract

The invention relates to an indoor scene reconstruction method based on pre-sampling and a progressive semantic modeling strategy. Comprising the following steps that a scene reconstruction model is constructed, and the scene reconstruction model comprises a DFormer module, a main model module and a semantic modeling module; training the constructed scene reconstruction model based on the NYUv2 data set; inputting the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model; the main model module carries out scene rendering pre-sampling on the input RGB image and the corresponding depth map, rough volume density estimation is obtained, and formal volume and color rendering is carried out; the DFormer module is used for converting the RGB image into a semantic image; the semantic modeling module takes a semantic image and a corresponding depth map as input, and assists in refining the volume and color rendering effect through a progressive semantic modeling strategy; and indoor scene reconstruction is realized. According to the method, the efficiency and precision of scene reconstruction can be remarkably improved, and more accurate detail reduction and a more efficient rendering process are realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and more specifically, to an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategies. Background Art

[0002] Indoor scene reconstruction is a technology that generates three-dimensional models from multi-view image data. It is widely used in virtual reality, augmented reality, architectural design and other fields. This technology focuses on the spatial structure and detail information of the scene. By capturing image information from different perspectives, computer vision and image processing methods are used to reconstruct the real indoor environment. Traditional reconstruction methods usually rely on multi-view geometry (MVG) or voxel-based three-dimensional modeling technology. Although these methods can provide high accuracy under certain conditions, they have high computational complexity, large storage requirements, and high requirements on the quality of input data. Especially when dealing with large-scale and complex indoor environments, they are prone to problems such as slow rendering speed and loss of details. In addition, since traditional methods mostly use explicit representations, such as voxel grids, such methods often seem to be unable to capture subtle structures and texture changes. In recent years, implicit neural representations (such as NeRF) have gradually become an effective three-dimensional reconstruction method. By implicitly storing scene information in a multi-layer perceptron (MLP), efficient modeling and rendering of three-dimensional scenes are achieved. Although such methods have significantly improved in accuracy, their demand for computing resources is still high, and the rendering efficiency is often limited by the processing speed of complex scenes. In addition, this type of method has limitations in scene understanding and is not effective in extracting information under occlusion and incomplete perspective.

[0003] In order to improve the efficiency and accuracy of scene reconstruction, it is necessary to introduce a variety of information to assist in scene understanding and modeling. Compared with traditional scene reconstruction methods, semantic information can help the model better identify and understand the distribution of objects in the scene, thereby effectively solving the problem of unclear edge information caused by blurred frames in RGB images. At the same time, the application of pre-sampling technology can quickly capture the overall structure of the scene and reduce computational complexity. Therefore, how to combine semantic information and pre-sampling technology has become the key to improving the efficiency and accuracy of scene reconstruction. Summary of the invention

[0004] The purpose of the present invention is to propose a method for indoor scene reconstruction based on pre-sampling and progressive semantic modeling strategies, aiming to solve the limitations of existing methods in reconstruction efficiency and accuracy. This method can quickly capture the overall structure of the scene, effectively reduce the computational complexity, and gradually optimize the scene rendering by progressively moving from coarse precision to fine precision, thereby improving the model's ability to understand complex scenes, thereby achieving efficient and accurate indoor scene reconstruction.

[0005] To achieve the above object, the present invention adopts the following technical solution: an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, comprising the following steps:

[0006] Step S1: constructing a scene reconstruction model, wherein the scene reconstruction model includes a DFormer module, a main model module and a semantic modeling module;

[0007] Step S2: A scene reconstruction model constructed based on the NYUv2 dataset training;

[0008] Step S3: input the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model;

[0009] The main model module performs scene rendering pre-sampling on the input RGB image and the corresponding depth map, obtains a rough volume density estimate, and performs formal volume and color rendering to complete the main reconstruction process of the scene;

[0010] The DFormer module converts the RGB image into a semantic image;

[0011] The semantic modeling module takes the semantic image and the corresponding depth map as input, and assists in refining the rendering effects of volume and color through a progressive semantic modeling strategy;

[0012] The indoor scene is reconstructed based on the output results of the main model module and the semantic modeling module.

[0013] Preferably, the main model module includes a sampling guided multilayer perceptron SG-MLP, a truncated signed distance field multilayer perceptron SDF MLP and a color multilayer perceptron RGB MLP; the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and the corresponding depth map, and performs a rough volume density estimation on the scene; the SDFMLP and RGB MLP of the main model module both use the output of the SG-MLP as input to perform independent rendering of volume and color respectively.

[0014] Preferably, the SG-MLP of the main model module renders and pre-samples the input RGB image and the corresponding depth map, and performs a rough volume density estimation on the scene, specifically, using the frequency encoding function γ of NeRF to pre-sample the sampling point set Each sampling point in the high frequency encoding is then high-frequency encoded, and the volume density of each sampling point after high frequency encoding is estimated and outputted as ρ i , the calculation formula is as follows:

[0015] ρ i =MLP θ (γ(z i ))

[0016] Among them, MLP θ represents the SG-MLP network, γ(x i ) is the sampling point z i The high-frequency coding of is used to enhance the spatial information through the frequency coding function γ in NeRF.

[0017] Preferably, the semantic modeling module includes a truncated signed distance field multilayer perceptron SDF MLP and a color multilayer perceptron RGB MLP independent of the main model module; the SDF MLP and RGB MLP of the semantic modeling module both take the semantic image and the corresponding depth map as input, and assist in refining the volume and color rendering effects through a progressive semantic modeling strategy.

[0018] Preferably, the progressive semantic modeling strategy is as follows:

[0019] The total number of rendering rounds, the coarse-precision voxel size, and the fine-precision voxel size are preset; the semantic image and the corresponding depth map are used as the input of the SDF MLP and RGB MLP in the coarse-precision rendering stage and the input of the SDF MLP and RGB MLP in the fine-precision rendering stage respectively; the coarse precision is first used to perform half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the coarse precision; then the fine precision is switched to perform the second half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the fine precision, wherein the light weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel in the coarse precision is used to calculate the light weight in the coarse-precision rendering stage, and the RGB value of each voxel in the coarse precision is used to calculate the RGB value in the fine-precision stage.

[0020] Preferably, the voxel size in the fine precision rendering stage is the same as the voxel size in the main model module rendering stage, and each dimension of the voxel in the coarse precision rendering stage is 10 times the voxel size in the fine precision rendering stage.

[0021] Preferably, the light weights of the coarse precision rendering stage are continued to the fine precision rendering stage, the SDF value of each voxel under coarse precision is used to calculate the light weights of the coarse precision rendering stage, and the RGB value of each voxel under coarse precision is used to calculate the RGB value of the fine precision stage, as follows:

[0022] In the coarse-precision rendering stage, the calculation of light weight depends on the volume density ρ of the sampling point. i , the light weight ω in the coarse precision stage coarse (k) Using NEUS weight calculation method:

[0023]

[0024] ρ j =σ(sdf j ·inv_s)

[0025] Among them, ω coarse (k) represents the light weight of the kth voxel in the coarse-precision rendering stage, ρ k Indicates the sampling point z corresponding to the k-th voxel k The volume density, Δz k =z k+1 -z k Represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage, ρ j Represents the SDF value sdf of the j-th voxel under coarse precision j The calculated volume density, inv_s is the scaling factor preset in the coarse precision rendering stage, and σ is the Sigmoid activation function;

[0026] In the fine-precision rendering stage, the light weight is calculated based on the weight in the coarse-precision rendering stage. The light weight ω in the fine-precision stage fine (l) The calculation formula is as follows:

[0027]

[0028] Among them, ω fine (l) represents the light weight of the lth voxel in the fine-precision rendering stage, ω coarse (l) represents the light weight of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, ρ l Indicates the sampling point z corresponding to the lth voxel l The volume density, Δz l =z l+1 -z l Indicates the distance between the sampling points corresponding to two adjacent voxels in the fine-precision rendering stage;

[0029] RGB value c of the fine-precision rendering stage fine The calculation of (l) is as follows:

[0030] c fine (l) = MLP rgb (c coarse (l))

[0031] Among them, c fine (l) is the RGB value of the lth voxel under fine precision, c coarse (l) is the RGB value of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, MLP rgb RGB MLP network at fine precision.

[0032] Preferably, the total loss function in the training of the scene reconstruction model is:

[0033]

[0034] in, is the loss of the main model module, is the loss of SG-MLP for density estimation, is the loss of the semantic modeling module, They are the color error loss, depth error loss and SDF value error loss between the actual and rendered main model modules. For the loss of free space outside the object, To constrain the smoothness loss of the signed distance field away from the surface, Smoothness loss to encourage continuity and smoothness of surface geometry; are the color error loss and SDF value error loss between the actual and rendered semantic modeling modules respectively; main , SG , sem , rgb , sdf , d , smo , eik , fs ,λ' rgb ,λ' sdf are the corresponding hyperparameters.

[0035] Preferably, the reconstruction of the indoor scene according to the output results of the main model module and the semantic modeling module is specifically:

[0036] The main model module calculates the SDF value, RGB value and light weight ω of each voxel main (l),generate a geometric representation of the scene;

[0037] The semantic modeling module determines the voxel size of the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel size of the main model module, and obtains the light weight ω of the fine-precision stage by calculation fine (l) SDF values ​​and RGB values ​​are used to improve rendering accuracy. The RGB values ​​obtained by fine-precision rendering are converted into semantic labels corresponding to each voxel through color mapping to enhance the semantic information representation of the scene.

[0038] The Marching Cubes algorithm is used to extract the mesh surface from the SDF value of each voxel in the fine-precision stage of the main model module and the semantic modeling module. The rendering results of the main model module and the semantic modeling module under fine precision are combined to generate a three-dimensional indoor scene mesh containing geometric structure and texture details, thereby realizing the reconstruction of the indoor scene.

[0039] Preferably, the rendering results of the main model module and the semantic modeling module are combined under fine precision, specifically:

[0040]

[0041] ρ m =σ(sdf m ·inv_s')

[0042] Among them I final Represents the final rendering result, ω main (l) is the light weight of the lth voxel calculated by the main model module, c l is the RGB value of the lth voxel in the main model module, L is the number of voxels, β is a hyperparameter, ρ m Represents the SDF value sdf of the mth voxel passing through the main model module m Calculated bulk density, ρ l Indicates the sampling point z corresponding to the lth voxel of the main model module l The volume density, Δz l =z l+1 -z l It represents the distance between the sampling points corresponding to two adjacent voxels in the main model module, inv_s' is the scaling factor preset in the main model module, and σ is the Sigmoid activation function.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The method proposed in this paper can significantly reduce the computational complexity through pre-sampling technology, and gradually optimize the scene rendering using a progressive semantic modeling strategy, thereby enhancing the model's understanding of the scene structure. This method enables the model to more accurately capture the detailed changes in the scene, gradually refine the volume and color features, and ultimately achieve a more efficient and accurate indoor scene reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the principle of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0047] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0049] The model of the present invention combines the advantages of pre-sampling and progressive semantic modeling, and efficiently utilizes semantic information in the rendering process to improve scene structure, detail understanding and reconstruction accuracy. The preliminary estimation of volume is achieved through SG-MLP, which effectively reduces the computational complexity, and a progressive semantic modeling strategy is adopted while rendering the main model to gradually capture rich volume and color features. Compared with traditional reconstruction methods, the present invention can restore scene details more accurately and achieve high-precision indoor scene reconstruction.

[0050] like Figure 1 As shown, this embodiment provides an indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, comprising the following steps:

[0051] Step S1: constructing a scene reconstruction model, wherein the scene reconstruction model includes a DFormer module, a main model module and a semantic modeling module;

[0052] Step S2: A scene reconstruction model constructed based on the NYUv2 dataset training;

[0053] Step S3: input the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model;

[0054] The main model module performs scene rendering pre-sampling on the input RGB image and the corresponding depth map, obtains a rough volume density estimate, and performs formal volume and color rendering to complete the main reconstruction process of the scene;

[0055] The DFormer module preprocesses the RGB image and converts the RGB image into a semantic image by combining the label mapping of the NYUv2 dataset in the pre-training stage;

[0056] The semantic modeling module uses the semantic image and the corresponding depth map as input, and assists in refining the rendering effects of volume and color through a progressive semantic modeling strategy, thereby assisting the main model in achieving more detailed scene restoration;

[0057] The indoor scene is reconstructed based on the output results of the main model module and the semantic modeling module.

[0058] In this embodiment, the main model module includes a sampling guided multilayer perceptron SG-MLP, a truncated signed distance field multilayer perceptron SDF MLP and a color multilayer perceptron RGB MLP; the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and the corresponding depth map, and performs a rough volume density estimation on the scene, and color rendering is not involved in this process; the SDF MLP and RGB MLP of the main model module both use the output of the SG-MLP as input to perform independent rendering of volume and color respectively.

[0059] In this embodiment, the SG-MLP of the main model module renders and pre-samples the input RGB image and the corresponding depth map, and makes a rough volume density estimation of the scene. Specifically, the frequency encoding function γ of NeRF is used to pre-sample the sampling point set. Each sampling point in the high frequency encoding is then high-frequency encoded, and the volume density of each sampling point after high frequency encoding is estimated and outputted as ρ i , the calculation formula is as follows:

[0060] ρ i =MLP θ (γ(z i ))

[0061] Among them, MLP θ represents the SG-MLP network, γ(x i ) is the sampling point z i The high-frequency coding of is used to enhance the spatial information through the frequency coding function γ in NeRF.

[0062] In this embodiment, the semantic modeling module includes a truncated signed distance field multilayer perceptron SDF MLP and a color multilayer perceptron RGB MLP independent of the main model module; the SDF MLP and RGB MLP of the semantic modeling module both take the semantic image and the corresponding depth map as input, and use the progressive semantic modeling strategy (PFPSMS) to assist in refining the rendering effects of volume and color, first using a coarse-precision feature plane for rapid preliminary rendering; then switching to a conventional fine-precision mode, using an independent MLP to perform fine rendering of the volume and color.

[0063] In this embodiment, the progressive semantic modeling strategy optimizes computational efficiency and reconstruction quality by gradually switching between coarse precision and fine precision, as follows:

[0064] The total number of rendering rounds, the coarse-precision voxel size, and the fine-precision voxel size are preset; the semantic image and the corresponding depth map are used as the input of the SDF MLP and RGB MLP in the coarse-precision rendering stage and the input of the SDF MLP and RGB MLP in the fine-precision rendering stage respectively; the coarse precision is first used to perform half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the coarse precision; then the fine precision is switched to perform the second half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the fine precision, wherein the light weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel in the coarse precision is used to calculate the light weight in the coarse-precision rendering stage, and the RGB value of each voxel in the coarse precision is used to calculate the RGB value in the fine-precision stage.

[0065] In this embodiment, the voxel size in the fine precision rendering stage is the same as the voxel size in the main model module rendering stage, and each dimension of the voxel in the coarse precision rendering stage is 10 times the voxel size in the fine precision rendering stage.

[0066] In this embodiment, the light weight of the coarse precision rendering stage is continued to the fine precision rendering stage, the SDF value of each voxel under coarse precision is used to calculate the light weight of the coarse precision rendering stage, and the RGB value of each voxel under coarse precision is used to calculate the RGB value of the fine precision stage, as follows:

[0067] In the coarse-precision rendering stage, the calculation of light weight depends on the volume density ρ of the sampling point. i , the light weight ω in the coarse precision stage coarse (k) Using NEUS weight calculation method:

[0068]

[0069] ρ j =σ(sdf j ·inv_s)

[0070] Among them, ω coarse (k) represents the light weight of the kth voxel in the coarse-precision rendering stage, ρ k Indicates the sampling point z corresponding to the k-th voxel k The volume density, Δz k =z k+1 -z k Represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage, ρ j Represents the SDF value sdf of the j-th voxel under coarse precision jThe calculated volume density, inv_s is the scaling factor preset in the coarse precision rendering stage, and σ is the Sigmoid activation function;

[0071] In the fine-precision rendering stage, the light weight is calculated based on the weight in the coarse-precision rendering stage. The light weight ω in the fine-precision stage fine (l) The calculation formula is as follows:

[0072]

[0073] Among them, ω fine (l) represents the light weight of the lth voxel in the fine-precision rendering stage, ω coarse (l) represents the light weight of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, ρ l Indicates the sampling point z corresponding to the lth voxel l The volume density, Δz l =z l+1 -z l Indicates the distance between the sampling points corresponding to two adjacent voxels in the fine-precision rendering stage;

[0074] RGB value c of the fine-precision rendering stage fine The calculation of (l) is as follows:

[0075] c fine (l) = MLP rgb (c coarse (l))

[0076] Among them, c fine (l) is the RGB value of the lth voxel under fine precision, c coarse (l) is the RGB value of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, MLP rgb RGB MLP network at fine precision.

[0077] In this embodiment, the total loss function in the training of the scene reconstruction model is:

[0078]

[0079] in, is the loss of the main model module, is the loss of SG-MLP for density estimation, is the loss of the semantic modeling module, They are the color error loss, depth error loss and SDF value error loss between the actual and rendered main model modules. For the loss of free space outside the object, To constrain the smoothness loss of the signed distance field away from the surface, Smoothness loss to encourage continuity and smoothness of surface geometry; are the color error loss and SDF value error loss between the actual and rendered semantic modeling modules respectively; main , SG , sem , rgb , sdf , d , smo , eik , fs ,λ' rgb ,λ' sdf is the corresponding hyperparameter. The calculations are all based on existing technologies, and the specific expressions are as follows:

[0080]

[0081] In this embodiment, the reconstruction of the indoor scene based on the output results of the main model module and the semantic modeling module is specifically as follows:

[0082] The main model module calculates the SDF value, RGB value and light weight ω of each voxel main (l),generate a geometric representation of the scene;

[0083] The semantic modeling module determines the voxel size of the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel size of the main model module, and obtains the light weight ω of the fine-precision stage by calculation fine (l) SDF values ​​and RGB values ​​are used to improve rendering accuracy. The RGB values ​​obtained by fine-precision rendering are converted into semantic labels corresponding to each voxel through color mapping to enhance the semantic information representation of the scene.

[0084] The Marching Cubes algorithm is used to extract the mesh surface from the SDF value of each voxel in the fine precision stage of the main model module and the semantic modeling module. The rendering results of the main model module and the semantic modeling module under fine precision are combined to optimize the geometric shape and texture information of the scene, generate a three-dimensional indoor scene mesh containing geometric structure and texture details, and realize the reconstruction of the indoor scene.

[0085] In this embodiment, the rendering results of the main model module and the semantic modeling module are combined in fine precision, specifically:

[0086]

[0087] ρ m =σ(sdf m ·inv_s')

[0088] Among them I final Represents the final rendering result, ω main (l) is the light weight of the lth voxel calculated by the main model module, c l is the RGB value of the lth voxel in the main model module, L is the number of voxels, β is a hyperparameter, ρ m Represents the SDF value sdf of the mth voxel passing through the main model module m Calculated bulk density, ρ l Indicates the sampling point z corresponding to the lth voxel of the main model module l The volume density, Δz l =z l+1 -z l It represents the distance between the sampling points corresponding to two adjacent voxels in the main model module, inv_s' is the scaling factor preset in the main model module, and σ is the Sigmoid activation function.

[0089] In summary, in view of the limited ability of existing methods to capture semantic and spatial details during scene reconstruction, as well as the low reconstruction efficiency and poor accuracy, a method for indoor scene reconstruction based on pre-sampling and progressive semantic modeling strategy is proposed. The model consists of a main model and a semantic modeling module. The main model uses SG-MLP for preliminary rendering pre-sampling to obtain the rough structural information of the scene, effectively reducing the computational complexity. The semantic modeling module adopts a progressive semantic modeling strategy to gradually fuse the feature planes of coarse precision and fine precision, quickly build a scene framework under coarse precision, and then switch to fine precision for fine rendering, so as to better capture the complex details and semantic features in the scene. Compared with traditional methods, the present invention can significantly improve the efficiency and accuracy of scene reconstruction, and achieve more accurate detail restoration and a more efficient rendering process.

[0090] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. An indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy, characterized by: The following steps are involved: Step S1: constructing a scene reconstruction model, wherein the scene reconstruction model includes a DFormer module, a main model module and a semantic modeling module; Step S2: A scene reconstruction model constructed based on the NYUv2 dataset training; Step S3: input the RGB image to be reconstructed and the corresponding depth map into the trained scene reconstruction model; The main model module performs scene rendering pre-sampling on the input RGB image and the corresponding depth map, obtains a rough volume density estimate, and performs formal volume and color rendering to complete the main reconstruction process of the scene; The DFormer module converts the RGB image into a semantic image; The semantic modeling module takes the semantic image and the corresponding depth map as input, and assists in refining the rendering effects of volume and color through a progressive semantic modeling strategy; The indoor scene is reconstructed based on the output results of the main model module and the semantic modeling module.

2. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 1 is characterized in that: The main model module includes a sampling guided multilayer perceptron SG-MLP, a truncated signed distance field multilayer perceptron SDF MLP and a color multilayer perceptron RGB MLP; the SG-MLP of the main model module performs rendering pre-sampling on the input RGB image and the corresponding depth map, and performs a rough volume density estimation on the scene; the SDF MLP and RGB MLP of the main model module both use the output of the SG-MLP as input to perform independent rendering of volume and color respectively.

3. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 2 is characterized in that: The SG-MLP of the main model module renders and pre-samples the input RGB image and the corresponding depth map, and makes a rough volume density estimate of the scene. Specifically, the frequency encoding function γ of NeRF is used to encode the sampling point set Each sampling point in the high frequency encoding is then high-frequency encoded, and the volume density of each sampling point after high frequency encoding is estimated and outputted as ρ i , the calculation formula is as follows: ρ i =MLP θ (γ(z i )) Among them, MLP θ represents the SG-MLP network, γ(x i ) is the sampling point z i The high-frequency coding of is used to enhance the spatial information through the frequency coding function γ in NeRF.

4. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 3 is characterized in that: The semantic modeling module includes a truncated signed distance field multilayer perceptron SDFMLP and a color multilayer perceptron RGB MLP, which are independent of the main model module; the SDF MLP and RGB MLP of the semantic modeling module both take the semantic image and the corresponding depth map as input, and assist in refining the rendering effects of volume and color through a progressive semantic modeling strategy.

5. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 4 is characterized in that: The progressive semantic modeling strategy is as follows: The total number of rendering rounds, the coarse-precision voxel size, and the fine-precision voxel size are preset; the semantic image and the corresponding depth map are used as the input of the SDF MLP and RGB MLP in the coarse-precision rendering stage and the input of the SDF MLP and RGBMLP in the fine-precision rendering stage respectively; the coarse precision is first used to perform half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the coarse precision; then the fine precision is switched to perform the second half of the rounds of volume and color rendering, and the SDF MLP and RGB MLP respectively output the SDF value and RGB value of each voxel in the fine precision, wherein the light weight of the coarse-precision rendering stage is continued to the fine-precision rendering stage, the SDF value of each voxel in the coarse precision is used to calculate the light weight in the coarse-precision rendering stage, and the RGB value of each voxel in the coarse precision is used to calculate the RGB value in the fine-precision stage.

6. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 5 is characterized in that: The voxel size in the fine precision rendering stage is the same as the voxel size in the main model module rendering stage, and each dimension of the voxel in the coarse precision rendering stage is 10 times the voxel size in the fine precision rendering stage.

7. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 5, characterized in that: The light weights of the coarse precision rendering stage are continued to the fine precision rendering stage. The SDF value of each voxel under coarse precision is used to calculate the light weights of the coarse precision rendering stage, and the RGB value of each voxel under coarse precision is used to calculate the RGB value of the fine precision stage, as follows: In the coarse-precision rendering stage, the calculation of light weight depends on the volume density ρ of the sampling point. i , the light weight ω in the coarse precision stage coarse (k) Using NEUS weight calculation method: r j =σ(sdf j ·inv_s) Among them, ω coarse (k) represents the light weight of the kth voxel in the coarse-precision rendering stage, ρ k Indicates the sampling point z corresponding to the k-th voxel k The volume density, Δz k =z k+1 -z k Represents the distance between the sampling points corresponding to two adjacent voxels in the coarse-precision rendering stage, ρ j Represents the SDF value sdf of the j-th voxel under coarse precision j The calculated volume density, inv_s is the scaling factor preset in the coarse precision rendering stage, and σ is the Sigmoid activation function; In the fine-precision rendering stage, the light weight is calculated based on the weight in the coarse-precision rendering stage. The light weight ω in the fine-precision stage fine (l) The calculation formula is as follows: Among them, ω fine (l) represents the light weight of the lth voxel in the fine-precision rendering stage, ω coarse (l) represents the light weight of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, ρ l Indicates the sampling point z corresponding to the lth voxel l The volume density, Δz l =z l+1 -z l Indicates the distance between the sampling points corresponding to two adjacent voxels in the fine-precision rendering stage; RGB value c of the fine-precision rendering stage fine The calculation of (l) is as follows: c fine (l)=MLP rgb (c coarse (l)) Among them, c fine (l) is the RGB value of the lth voxel under fine precision, c coarse (l) is the RGB value of the voxel in the coarse-precision rendering stage corresponding to the l-th voxel in the fine-precision rendering stage, MLP rgb RGB MLP network at fine precision.

8. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 5, characterized in that: The total loss function in the training of the scene reconstruction model is: in, is the loss of the main model module, is the loss of SG-MLP for density estimation, is the loss of the semantic modeling module, They are the color error loss, depth error loss and SDF value error loss between the actual and rendered main model modules. For the main model module, the loss of free space outside the object is To constrain the smoothness loss of the signed distance field away from the surface, Smoothness loss to encourage continuity and smoothness of surface geometry; are the color error loss and SDF value error loss between the actual and rendered semantic modeling modules, respectively, main , SG , sem , rgb , sdf , d , smo , eik , fs ,λ′ rgb ,λ′ sdf are the corresponding hyperparameters.

9. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 1, characterized in that: The reconstruction of the indoor scene based on the output results of the main model module and the semantic modeling module is specifically as follows: The main model module calculates the SDF value, RGB value and light weight ω of each voxel main (l),generate a geometric representation of the scene; The semantic modeling module determines the voxel size of the coarse-precision rendering stage and the fine-precision rendering stage based on the voxel size of the main model module, and obtains the light weight ω of the fine-precision stage by calculation fine (l), SDF value and RGB value, used to improve rendering accuracy; The RGB values ​​obtained by fine-precision rendering are converted into semantic labels corresponding to each voxel through color mapping to enhance the semantic information representation of the scene; The Marching Cubes algorithm is used to extract the mesh surface from the SDF value of each voxel in the fine-precision stage of the main model module and the semantic modeling module. The rendering results of the main model module and the semantic modeling module under fine precision are combined to generate a three-dimensional indoor scene mesh containing geometric structure and texture details, thereby realizing the reconstruction of the indoor scene.

10. The indoor scene reconstruction method based on pre-sampling and progressive semantic modeling strategy according to claim 9, characterized in that: The combination of the main model module and the rendering results of the semantic modeling module under fine precision is specifically as follows: r m =σ(sdf m ·inv_s′) Among them I final Represents the final rendering result, ω main (l) The light weight of the ,th voxel calculated by the main model module, c l is the RGB value of the voxel in the main model module, L is the number of voxels, β is a hyperparameter, ρ m Represents the SDF value sdf of the mth voxel through the main model module m Calculated bulk density, ρ l Indicates the sampling point z corresponding to the lth voxel of the main model module l The volume density, Δz l =z l+1 -z l It represents the distance between the sampling points corresponding to two adjacent voxels in the main model module, inv_s' is the scaling factor preset in the main model module, and σ is the Sigmoid activation function.

Images