Multi-resolution neural radiance field method for multi-scale efficient and high-precision digital twinning of building groups

Abstract

The application provides a multi-resolution neural radiation field method for multi-scale efficient high-precision digital twinning of a building group. The method comprises the following steps: step 1, multi-view apparent image acquisition and multi-resolution dataset generation of the building group; step 2, establishment of a multi-resolution neural network model for scene rendering color and volume density estimation of the building group; step 3, training of a multi-resolution neural network model for scene rendering and three-dimensional reconstruction of the building group by using a super-resolution enhancement volume rendering technology; and step 4, multi-scale scene rendering and new view synthesis of the building group by using the trained multi-resolution neural network model. The method realizes arbitrary view image rendering, efficient high-precision three-dimensional reconstruction and digital twinning of a multi-scale large-scale building group scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction, scene rendering, digital twins, and intelligent operation and maintenance of building structures. Specifically, it relates to a multi-resolution neural radiation field method for efficiently and accurately creating multi-scale digital twins of building complexes. This method addresses the challenges of multi-resolution, high-precision 3D modeling and detailed rendering for multi-scale building complexes. It has broad applications in building complex visualization and virtual reality, structural health monitoring and disaster prevention and mitigation, architectural design and planning, and intelligent construction. Background Art

[0002] 3D reconstruction technology aims to restore the 3D structure of a scene from multi-view 2D images. Traditional 3D reconstruction technology is based on geometric methods, including multi-view stereo vision (MVS) and laser scanning, and achieves high-precision 3D reconstruction through explicit modeling. The following challenges exist in using these traditional methods to perform surface damage mapping and real-time updating of 3D models of building complex structures: (1) lack of multi-scale processing and automated editing for building complex scenes, (2) highly restricted by image feature matching accuracy, and poor performance for target areas with sparse or repeated textures, (3) heavy reliance on manual operation and professional modeling software, insufficient intelligence, low efficiency and capabilities, and (4) the need for high-performance sensing equipment (such as 3D laser point cloud scanners, etc.).

[0003] In recent years, deep learning-driven implicit modeling methods such as Neural Radiance Field (NeRF) have achieved high-quality new perspective image generation by directly fitting the radiation field function of a three-dimensional scene through a neural network. The advantages of the NeRF method in high-quality three-dimensional reconstruction and perspective-independent real-time rendering provide potential possibilities for three-dimensional reconstruction, scene rendering, and detail presentation in complex scenes of building complexes, and bring new ideas for the realization of multi-scale three-dimensional reconstruction of building complexes, structural damage mapping, and real-time digital twins. However, the NeRF method still has problems such as high computational complexity, insufficient expression of details, and poor adaptability to large-scale scenes. However, these inherent problems become particularly prominent when it comes to the three-dimensional reconstruction of large-scale building complexes, posing new challenges for the realization of efficient and high-precision three-dimensional reconstruction and digital twin modeling of multi-scale building complexes:

[0004] (1) Building complex visual data (multi-view images or videos) contains multi-scale information, while traditional NeRF methods often only have good performance for a single small scale;

[0005] (2) The digital twin of a building complex has a multi-resolution modeling requirement. That is, when the details are not important, only the coarse-scale modeling results need to be presented, while when the details are important, the fine-scale modeling results need to be presented. However, the traditional NeRF method can only model at a single scale and cannot adapt to the automatic adjustment requirements of multiple resolutions.

[0006] (3) If the traditional NeRF method is directly used to deal with the problem of modeling large-scale, detailed building complex scenes, there will be problems such as poor detail feature capture ability (low accuracy) of the single neural network implicit proxy model and long rendering time (low efficiency). Summary of the Invention

[0007] This paper aims to address the existing technical issues by proposing a multi-resolution neural radiation field method for efficient and high-precision digital twinning of multi-scale building complexes. This method enables arbitrary-viewpoint image rendering, efficient and high-precision 3D reconstruction, and digital twinning of multi-scale, large-scale building complexes.

[0008] The present invention is achieved through the following technical solutions. The present invention proposes a multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes. The method includes the following steps:

[0009] Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets;

[0010] Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation;

[0011] Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction;

[0012] Step 4: Use the trained multi-resolution neural network model to render multi-scale building complex scenes and synthesize new views.

[0013] Furthermore, the step 1 specifically includes the following steps:

[0014] Step 1: Collect multi-view apparent image data of the building complex, and convert the single-resolution image of the target scene into a multi-resolution image through the image super-resolution generation model. For the collected image data, first convert the original resolution image into a high-resolution image through the image super-resolution model SR, that is, Represents the high-resolution image after super-resolution, SR is the image super-resolution pre-training generation model adopted; then Downsampling divides image data into different resolutions Right now:

[0015]

[0016] Among them, Subsample represents the downsampling operation of the image, r represents the resolution level of the image, and k is the highest resolution image The resolution level;

[0017] Step 1 and 2: Estimate the parameters of the image data to obtain the camera's internal and external parameters, and calculate the transformation matrix from camera coordinates to world coordinates and the camera's relative position and pose;

[0018] Conversion from pixel coordinate system to camera coordinate system:

[0019]

[0020] Where, is the camera coordinate system, O uv is the image pixel coordinate system, K is the camera’s intrinsic parameter matrix, f x , f y are the focal lengths of the camera in the x and y directions, (c x , c y ) is the principal point coordinate of the camera;

[0021] Coordinate transformation from camera coordinate system to world coordinate system:

[0022]

[0023] Where, is the world coordinate system, c2w is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix R i and the translation vector T i .

[0024] Furthermore, the step 2 specifically includes the following steps:

[0025] Step 21: Perform discrete light segment sampling to mark the key feature target area;

[0026] Step 22: Take images of different resolutions as input, embed the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding, and the image appearance code to form a coding vector containing information of different scales;

[0027] Step 2 and 3: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene of the building complex; k The spatial coordinates P(x, y, z), the normalized direction vector d and the image appearance embedding are used as input to establish a multi-resolution neural network model, which outputs the color value c and volume density value σ of each sampling point.

[0028] Furthermore, the step 21 specifically includes:

[0029] (1) Camera ray generation based on normalized direction vector;

[0030] (2) Marking of key feature target areas;

[0031] (3) importance sampling of discrete rays in a general area;

[0032] (4) Optimization of discrete light segment sampling for key feature target areas.

[0033] Furthermore, the step 22 specifically includes:

[0034] (1) Multi-resolution hash coding of position information; specifically:

[0035] Step 1: Normalize spatial coordinates;

[0036] Step 2: Determine the hash-coded multi-resolution levels;

[0037] Step 3: Hash the vertices of the voxel grid at L resolution levels;

[0038] Step 4: Linearly interpolate the feature vector to match the input coordinates;

[0039] (2) Spherical harmonic function encoding of direction information: First, normalize the direction vector Convert to spherical coordinate direction vector The direction information of the sampling point is then encoded through a spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding, thereby realizing directional feature embedding. The specific calculation steps are as follows:

[0040]

[0041] Where l is the order of the spherical harmonics, m is the sequence of the spherical harmonics and -l<m<1, is the associated Legendre polynomial, is the normalization factor;

[0042] (3) Image appearance encoding: An additional neural network layer Embedding layer is embedded before the input of the color value prediction module. This layer maps each input image to a low-dimensional appearance encoding vector AppearanceEmbeddings.

[0043] Furthermore, the step three specifically includes:

[0044] Step 31: Generate pixel color prediction values ​​of the multi-view image of the building complex using volume rendering technology;

[0045] Step 32: Generate a high-resolution enhanced image through the super-resolution module;

[0046] Step 33: Extract feature edges or target areas through edge detection or semantic segmentation algorithms, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume-rendered feature edge map or target area map;

[0047] Steps 3 and 4: Calculate the weighted sum of the L2 norm loss and train the color and volume density estimation neural network for building scene rendering and 3D reconstruction in a supervised learning manner.

[0048] Furthermore, in steps three and four, the rendered image is converted into a high-contrast feature edge map or a target area map, which is compared with the true value of the high-contrast feature edge map or the target area map, and the loss between the generated image and the true value is calculated; the loss function The L2 norm loss is used and the calculation method is:

[0049]

[0050] Where, is the volume rendering loss, is the super-resolution loss, is the edge enhancement loss; ω1, ω2, and ω3 are weighted coefficients of different loss function components; Ω1 is the set of all sampled pixels from the model training image dataset, and Ω2 is the set of all sampled pixels from the high-resolution image dataset; C low (r) is the color value of the known pixel on the low-resolution image, is the pixel color value of the model body rendering image, C high (r) is the color value of the known pixel on the high-resolution image, The pixel color value of the image generated by the super-resolution module, C edge (r) is the color value of the pixel in the focus area of ​​the model training image, The color value of the pixel in the focus area of ​​the model body rendering image.

[0051] Furthermore, the step 4 specifically includes:

[0052] (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and sample on the rays;

[0053] (2) The spatial coordinates P (x, y, z) of the sampling point and the direction vector d are used as input to the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value c (s) of each sampling point is output. k ) and volume density value σ;

[0054] (3) Combining volume rendering and super-resolution technology to predict the color values ​​of image pixels at different resolutions, and realize the generation of multi-scale new perspective images.

[0055] The present invention also proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0056] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The light sampling module has been optimized. By combining segmented sampling and enhanced sampling in key areas, the sampling density is dynamically adjusted to achieve more refined high-resolution rendering modeling for areas with rich details such as building edges, corners, and textures, thereby achieving high-precision improvement.

[0059] (2) Through the fusion of multi-resolution hash coding spatial features, global structures and local details are captured simultaneously at different spatial scales, enhancing the representation ability and rendering accuracy of multi-scale 3D reconstruction scenes of building complexes and achieving multi-resolution improvement;

[0060] (3) Introducing a super-resolution module to significantly improve image clarity and accuracy, enhance the model's detailed representation of textures, edges, and fine structures, avoid the high cost of direct high-resolution rendering modeling, and achieve high accuracy and efficiency improvements;

[0061] (4) Design a small multi-layer perceptron (MLP) architecture for hash coding, volume density prediction, and color value prediction modules. By using hash coding, the computational efficiency is significantly improved and memory consumption is reduced. At the same time, the small MLP also reduces model parameters and computational complexity, thereby accelerating the training and inference process, improving the efficiency of real-time 3D modeling and scene rendering, and achieving high efficiency improvement.

[0062] (5) It is applicable to multi-view images or video visual data of building complexes containing multi-scale information, overcoming the limitation of traditional NeRF methods that often only model small scales;

[0063] (6) It realizes multi-resolution modeling of digital twins of building complexes. That is, when details are not important, only coarse-scale modeling results are presented, and when details are important, fine-scale modeling results are presented. This breaks through the limitation of the traditional NeRF method that can only model at a single scale and cannot adapt to multi-resolution automatic adjustment.

[0064] (7) By designing efficient computing modules for hash coding, volume density prediction, and color value prediction, the problems of large size and long rendering time of single neural network implicit proxy model in directly using traditional NeRF method to deal with large-scale building complex modeling are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0066] Figure 1 It is a flow chart of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes.

[0067] Figure 2 This is a schematic diagram of the architecture of the super-resolution generation model for multi-view appearance images of building complexes.

[0068] Figure 3 This is a schematic diagram of the initial segmented sampling process of discrete rays in a general area.

[0069] Figure 4 Schematic diagram of the discrete ray importance sampling process for a general area.

[0070] Figure 5 It is a schematic diagram of the discrete light enhancement sampling process considering the marking of the target area of ​​key features of the building complex.

[0071] Figure 6 Schematic diagram of a multi-resolution hash encoder for position information.

[0072] Figure 7 Schematic diagram of the multi-resolution neural radiance field model architecture for color and volume density estimation of three-dimensional scenes of building complexes.

[0073] Figure 8 This is a schematic diagram of the architecture of the two-dimensional image super-resolution enhancement module of the building complex neural radiation field model.

[0074] Figure 9 It is a schematic diagram of scene rendering and new view synthesis based on the pre-trained 3D reconstruction model of the building complex. DETAILED DESCRIPTION

[0075] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0076] Combine Figures 1-9 The present invention proposes a multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes, which includes the following steps:

[0077] Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets;

[0078] The step 1 specifically includes the following steps:

[0079] Step 1: Collect multi-view apparent image data of the building complex, and convert the single-resolution image of the target scene into a multi-resolution image through image super-resolution generation model processing; use a high-resolution camera to collect 360° full-view panoramic image data of the target scene building complex (denoted as I i , where i is the image number), to obtain comprehensive surface information of the building complex. For the collected image data, the original resolution image is first converted into a high-resolution image through the image super-resolution model (SR), that is, Represents the high-resolution image after super-resolution, SR is the image super-resolution pre-training generation model adopted; then Downsampling divides image data into different resolutions Right now:

[0080]

[0081] Among them, Subsample represents the downsampling operation of the image, r represents the resolution level of the image, and k is the highest resolution image The resolution level;

[0082] Select specific high-resolution image data levels based on application requirements As model training input (m represents the selected r value, ranging from 1, 2, ..., k), and a specific high-resolution image and specific low-resolution image data As a benchmark dataset for calculating model loss, the dataset represents architectural scenes at different resolution scales through super-resolution and downsampling of image data. Lower-resolution images focus on the overall representation of architectural scenes, while high-resolution images ensure that detailed features of buildings (such as building edges and building surface textures) are fully captured and preserved.

[0083] Step 1 and 2: Estimate the parameters of the image data to obtain the camera's internal and external parameters, and calculate the transformation matrix from camera coordinates to world coordinates and the camera's relative position and pose;

[0084] Select the high-resolution image set As data input, image processing tools (such as COLMAP) are used to estimate the parameters of the collected building scene image data, obtaining the camera intrinsic parameters (camera intrinsic parameter matrix K), camera extrinsic parameters (rotation matrix R and translation vector T), and the sparse reconstruction results of the scene. Then, combining the camera intrinsic and extrinsic parameters with the original 2D image information, the 2D coordinates of the image information are converted to 3D coordinates in the world coordinate system. The specific steps are as follows:

[0085] Conversion from pixel coordinate system to camera coordinate system:

[0086]

[0087] Where, is the camera coordinate system, O uv is the image pixel coordinate system, K is the camera’s intrinsic parameter matrix, f x , f y are the focal lengths of the camera in the x and y directions, (c x , c y ) are the principal point coordinates of the camera (the center of the image);

[0088] Coordinate transformation from camera coordinate system to world coordinate system:

[0089]

[0090] Where, is the world coordinate system, c2w is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix R i and the translation vector T i .

[0091] Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation;

[0092] The step 2 specifically includes the following steps:

[0093] Step 21: Perform discrete light segment sampling to mark the key feature target area; specifically including:

[0094] (1) Camera ray generation based on normalized direction vector:

[0095] Subtract the optical center coordinate (c x , c y ), get the absolute direction vector (ic) in homogeneous coordinates x , jc y , f), normalize the focal length f as the z-axis coordinate division to obtain the normalized direction vector under homogeneous coordinates Based on the c2w transformation matrix obtained in steps 1 and 2, the camera center coordinates and the normalized direction vector are transformed into the world coordinate system to form a ray r(t) = o + td, where o represents the optical center, t represents the distance along the direction of the ray, and d represents the normalized direction vector.

[0096] (2) Marking of key feature target areas:

[0097] The edge feature map or target area map is extracted from the original high-resolution image through edge detection or semantic segmentation algorithm, and the image edge or target area is marked as high value and the background area is marked as low value to obtain a high-contrast enhanced sampling image (denoted as The edge detection algorithm is used to enhance the high-contrast areas in the sampled image, representing areas rich in detail such as building edges, corners, and textures. The semantic segmentation algorithm is used to extract the pixel sets where the main buildings in the scene are located. These are the main target areas for 3D reconstruction of the buildings. These marked targets will be processed in the subsequent discrete ray segmentation sampling process.

[0098] (3) Importance sampling of discrete rays in a general area:

[0099] The camera ray generation and discrete ray sampling process based on the normalized direction vector is as follows Figure 3 As shown. Discretize along the direction of the light to obtain a finite number of sampling points, and use the importance sampler to sample a set of orthogonal points between the near boundary near and the far boundary far along the light. Among them, o is the optical center, c is the corresponding coordinate of the optical center on the imaging plane, s1~s n is the sampling point on the ray r(s)=o+td. The discrete ray importance sampler of the general area is as follows Figure 4 As shown, the specific sampling steps are:

[0100] Step 1: Preliminary piecewise sampling (Piecewise Sampling).

[0101] At a fixed distance near the camera [t n , t m], a uniform sampling strategy is used to generate sample points s with consistent intervals i ; The longest distance t beyond uniform sampling m Finally, the linear sampling strategy (LinearSampling) is used, and the sample step size increases linearly with the increase of distance, and the sampling is continued until the farthest distance from the camera t f Such a combined sampling strategy allows for efficient sampling of distant objects while still maintaining a dense set of samples for nearby objects. The sampling process can be expressed as:

[0102] s i ~Uniform(t n , t m ), i=1,2,...,N1 (4)

[0103] s j =t m +(j·Δs),t f =max(s j ), j=1,2,…,N2 (5)

[0104] Among them, s i Indicates the position of the ith sample near the camera, Uniform indicates the position of the ith sample near the camera. n , t m ] uniform sampling within s j represents the position of the jth sample at the far camera end to the near camera end, and Δs is the step size that increases as j increases.

[0105] Step 2: Proposal Network Sampling.

[0106] The main goal of the proposal network sampling is to integrate the sample position into the area of ​​the scene that contributes most to the final rendering, usually the first surface intersection. First, a small fusion multi-layer perceptron (MLP) and hash code are used to represent the density function of the scene:

[0107] σ(s)=DensityFild(hash(s)) (6)

[0108] According to the density σ(s) predicted by the proposal network, a probability density function (PDF) sampler is constructed to generate a new set of samples. The total number of these new samples (N=N1+N2) remains unchanged and is located at the position existing in the predicted scene:

[0109] s~PDF(σ(s)) (7)

[0110] Where s′ is the resampled sample point location, and PDF(σ(s)) is the probability density function based on the predicted density σ(s). The proposed network sampler can also be chained with multiple density functions to further integrate sampling. This invention uses two density functions; using more than two may result in diminishing returns.

[0111] (4) Optimization of discrete light segment sampling for key feature target areas

[0112] In addition to sampling enhancement on the same ray, since different rays representing the scene contain inconsistent target information, an optimization strategy is also used for sampling different rays. Figure 5 As shown. For (2) the high contrast enhanced sampling image obtained by marking the key feature target area Perform pixel value weighting to obtain sampling weight maps for different rays. When sampling each ray, multiply the total number of sampling points N on the ray by the corresponding weight e in the ray sampling weight map. j , get the total number of new light sampling points N′=e j N. If the light intersects the key area, the weight value e j If it is greater than 1, the sampling density of the light is increased, and the additional sample points are evenly interpolated in the original sample point distribution; if the light passes through a non-key area, the weight value e j Equal to 1, that is, the original sampling density of the light is used.

[0113] Step 22: Take images of different resolutions as input, embed the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding, and the image appearance code to form a coding vector containing information of different scales; the coding process specifically includes:

[0114] (1) Multi-resolution hash coding of position information

[0115] The spatial coordinates of the sampling points are encoded through a hash encoder to achieve position feature embedding and obtain the image sampling point position feature vector based on multi-resolution hash coding. Taking the multi-resolution hash encoder of position information in two-dimensional space as an example, Figure 6 The specific steps are as follows.

[0116] Step 1: Normalize spatial coordinates

[0117] First, normalize the dimensions of the input spatial coordinates p = (x, y, z) to the range [0, 1]:

[0118]

[0119] Where min(x) and max(x) are the minimum and maximum values ​​of the original space coordinates respectively.

[0120] Step 2: Determine the hash-encoded multi-resolution levels

[0121] The sampling space is divided into different levels of detail L according to different resolutions. i Divided into several uniform spatial grids, each resolution level L i There are k hash coding levels corresponding to a different hash encoder using a special hash function. Each level L i The number of grid points N i and the common eigenvector dimension F. The number of grid points N i By level L i Increase and decrease to capture feature information of different scales, which can be set as a geometric series, such as N i =N min b i-1 , where N min is the minimum number of grid points, and b is the scaling factor.

[0122] Step 3: Hash the vertices of the voxel grid at L resolution levels.

[0123] Use a special hash function to convert the three-dimensional grid coordinates p grid Mapping to hash values ​​h at different resolution levels i , and then look up the hash code vector from the pre-set hash table:

[0124]

[0125] Among them, v i is the hash code vector (whose dimension is F) mapped by the grid space coordinates in the hash table, Hash functions for implementing simple integer mappings (such as linear congruential generators).

[0126] Step 4: Linearly interpolate the feature vectors to match the input coordinates.

[0127] For each resolution level L i , the normalized spatial coordinates p′ of any point are mapped to the grid index, grid , in Indicates rounding down, grid index I grid Indicates the normalized space coordinate P′ in the Lth i The grid position to which the level belongs. By grid index I gridDetermine the nearest voxel grid around the spatial coordinates, use the hash table to obtain the encoding vector for weighted interpolation, and concatenate the feature codes of all resolution levels to obtain the position code of any sampling point. The specific calculation steps are as follows:

[0128]

[0129] in, is the hash code vector of the nearest voxel grid around the spatial coordinate p′, w k is the interpolation weight of each voxel grid, v′ i is the spatial coordinate p′ at the resolution level L i The hash code vector under φ(p) represents the position code vector of any sampling point p′ in the space obtained by concatenating the feature codes at all resolution levels.

[0130] (2) Spherical harmonic encoding of directional information

[0131] First, normalize the direction vector Convert to spherical coordinate direction vector The direction information of the sampling point is then encoded through a spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding, thus realizing the embedding of direction features. The specific calculation steps are as follows:

[0132]

[0133] Where l is the order of the spherical harmonics, m is the sequence of the spherical harmonics and -l<m<1, is the associated Legendre polynomial, is the normalization factor.

[0134] (3) Image appearance coding

[0135] An additional neural network layer (Embedding layer) is embedded before the input of the color value prediction module. This layer maps each input image to a low-dimensional appearance encoding vector (Appearance Embeddings). By embedding learnable appearance encoding vectors, the model optimizes an appearance encoding vector for each training image during training. This captures the image's specific appearance characteristics, helping the model correct for exposure differences in different shooting conditions during rendering, thereby generating more consistent and realistic view synthesis results.

[0136] Step 2 and 3: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene of the building complex; kThe spatial coordinates P(x, y, z), the normalized direction vector d and the image appearance embedding are used as input to establish a multi-resolution neural network model, which outputs the color value c and volume density value σ of each sampling point. The model architecture is as follows Figure 7 As shown, it includes a position information embedding module, a direction information embedding module, an appearance coding embedding module, a volume density prediction module, and a color value prediction module.

[0137] (1) Position information embedding module: that is, the position information multi-resolution hash coding module in step 22.

[0138] (2) Direction information embedding module: that is, the spherical harmonic function encoding module of the direction information in step 22.

[0139] (3) Appearance coding embedding module: the embedding network module of the image appearance coding in step 22

[0140] (4) Volume density prediction module: It uses a small multi-layer perceptron (MLP), i.e., a multi-layer neural network structure, and inputs a feature vector φ(p) containing hash codes of the spatial positions of sampling points at different levels, and outputs the spatial feature vector and the volume density prediction value σ corresponding to the spatial coordinates.

[0141] (5) Color value prediction module: It uses a small multi-layer perceptron MLP, i.e., a multi-layer neural network structure. Its input includes: the sampling point direction feature vector SH(d), the spatial feature vector output by the volume density prediction module, and the appearance embedding encoding vector of the original image (Appearance Embeddings). It outputs the color scalar prediction value c(C k ).

[0142] Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction;

[0143] The step three specifically includes:

[0144] Step 31: Generate pixel color prediction values ​​of the multi-view image of the building complex using volume rendering technology;

[0145] Step 3: Use volume rendering technology to generate pixel color prediction values ​​of the multi-view image of the building complex, and generate a low-resolution image through discretized volume rendering technology. The specific steps are as follows:

[0146] Extract N′ sample points t from the light along the light path according to the sampling strategy in step 21 i , perform weighted summation on the color contributions of the selected sampling points to generate the predicted pixel color value. The weighted summation calculation method for volume rendering discretization is:

[0147]

[0148] Where, t i Represents the t value corresponding to the i-th sampling point after discretization on the light, δ i Represents the distance between adjacent sampling points, T i is the transparency function from the viewpoint to the i-th sampling point, α i is the predicted value of the volume density of the i-th sampling point, c i is the color prediction value of the i-th sampling point, Represents the color prediction value of the image pixel after discretization volume rendering.

[0149] Step 32: Generate a high-resolution enhanced image through the super-resolution module;

[0150] Step 3.2 converts the low-resolution image generated in step 3.1 into a high-resolution enhanced image through the super-resolution module, reducing the computational overhead and training time of multi-resolution modeling. The module architecture is as follows: Figure 8 As shown, specifically including:

[0151] (1) The super-resolution module is used as an image enhancement module in the form of a pre-trained super-resolution model, and does not participate in the training and parameter optimization of the neural radiation field model, further reducing the computational cost.

[0152] (2) The training data used by the super-resolution model includes downsampled images of the original high-resolution images and low-resolution images generated by volume rendering. The super-resolution module is used to enhance the low-resolution rendering results and super-resolve the images to the same resolution level as the highest-resolution images.

[0153] (3) In addition, some parameters of the super-resolution model can be fine-tuned during the radiation field model training process. Although the amount of calculation increases slightly, the fine-tuned super-resolution model can make targeted optimizations for the image information of the target scene of the building complex, thereby obtaining better modeling effects.

[0154] Step 33: Extract feature edges or target areas through edge detection or semantic segmentation algorithms, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume-rendered feature edge map or target area map;

[0155] Steps 3 and 4: Calculate the weighted sum of the L2 norm loss and train the color and volume density estimation neural network for building scene rendering and 3D reconstruction in a supervised learning manner.

[0156] Steps 3 and 4 calculate the model training loss to test the model's rendering effect on the key areas of the building scene and the model generation effect on building scene views of different resolution scales. Specifically, they include:

[0157] (1) Convert the rendered image into a high-contrast feature edge map or target area map, compare it with the true value of the high-contrast feature edge map or target area map obtained in step 2.1, and calculate the loss between the generated image and the true value. Loss function The L2 norm loss is used and the calculation method is:

[0158]

[0159] Where, is the volume rendering loss, is the super-resolution loss, is the edge enhancement loss; ω1, ω2, and ω3 are weighted coefficients of different loss function components; Ω1 is the set of all sampled pixels from the model training image dataset, and Ω2 is the set of all sampled pixels from the high-resolution image dataset; C low (r) is the color value of the known pixel on the low-resolution image, is the pixel color value of the model body rendering image, C high (r) is the color value of the known pixel on the high-resolution image, The pixel color value of the image generated by the super-resolution module, C edge (r) is the color value of the pixel in the focus area of ​​the model training image, The color value of the pixels in the focus area of ​​the model body rendering image.

[0160] (2) The neural network model is trained in a supervised learning manner, using a gradient descent-based optimization algorithm (such as Adam, RAdam, SGD, etc.) to ensure that the color prediction value of the pixel point in the final super-resolution image is as close as possible to the true color value of the corresponding pixel point in the original high-resolution building scene image. At the same time, the loss between the volume rendering image and the real image and the influence of the sampling of the key attention area are also considered.

[0161] Step 4: Use the trained multi-resolution neural network model to render multi-scale building complex scenes and synthesize new views.

[0162] Step 4 uses the trained multi-resolution neural network to render multi-scale building scene and synthesize new views. The specific process is as follows: Figure 9 As shown, including:

[0163] (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and sample on the rays;

[0164] (2) The spatial coordinates P (x, y, z) of the sampling point and the direction vector d are used as input to the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value c (s) of each sampling point is output.k ) and volume density value σ;

[0165] (3) Combining volume rendering and super-resolution technology to predict the color values ​​of image pixels at different resolutions, and realize the generation of multi-scale new perspective images.

[0166] The present invention also proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0167] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0168] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0169] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0170] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0171] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0172] The above is a detailed introduction to the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes, characterized by: The method comprises the following steps: Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets; The step 1 specifically includes the following steps: Step 1: Collect multi-view apparent image data of the building complex, and convert the single-resolution image of the target scene into a multi-resolution image through the image super-resolution generation model. For the collected image data, first convert the original resolution image into a high-resolution image through the image super-resolution model SR, that is, , represents the high-resolution image after super-resolution, The model is pre-trained for the adopted image super-resolution; then the high-resolution image is Downsampling divides image data into different resolutions ;Right now: in, Indicates downsampling of the image. Indicates the resolution level of the image. For the highest resolution image The resolution level; Step 1 and 2: Estimate the parameters of the image data to obtain the camera's internal and external parameters, and calculate the transformation matrix from camera coordinates to world coordinates and the camera's relative position and pose; Conversion from pixel coordinate system to camera coordinate system: Where, is the camera coordinate system, is the image pixel coordinate system, K is the intrinsic parameter matrix of the camera, The camera is and The focal length in the direction, ( ) is the principal point coordinate of the camera; Coordinate transformation from camera coordinate system to world coordinate system: Where, is the world coordinate system, is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix and translation vectors ; Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation; The step 2 specifically includes the following steps: Step 21: Perform discrete light segment sampling to mark the key feature target area; Step 22: Take images of different resolutions as input, embed the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding, and the image appearance code to form a coding vector containing information of different scales; Step 2 and 3: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene of the building complex; The spatial coordinates of , normalized direction vector And image appearance embedding as input, build a multi-resolution neural network model, and output the color value of each sampling point and body density values ; Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction; The step three specifically includes: Step 31: Generate pixel color prediction values ​​of the multi-view image of the building complex using volume rendering technology; Step 32: Generate a high-resolution enhanced image through the super-resolution module; Step 33: Extract feature edges or target areas through edge detection or semantic segmentation algorithms, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume-rendered feature edge map or target area map; Steps 3 and 4: Calculation L A weighted sum of 2-norm losses is used to train color and volume density estimation neural networks for building scene rendering and 3D reconstruction in a supervised learning manner; Step 4: Use the trained multi-resolution neural network model to render multi-scale building scene and synthesize new views; The step 4 specifically includes: (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and sample on the rays; (2) The spatial coordinates of the sampling points , direction vector As input, it is input into the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value of each sampling point is output. and body density values ; (3) Combining volume rendering and super-resolution technology to predict the color values ​​of image pixels at different resolutions, and realize the generation of multi-scale new perspective images.

2. The method according to claim 1, characterized in that The step 21 specifically includes: (1) Camera ray generation based on normalized direction vectors; (2) Marking of key feature target areas; (3) Importance sampling of discrete rays in a general area; (4) Optimization of discrete light segment sampling for key feature target areas.

3. The method according to claim 2, characterized in that The step 22 specifically includes: (1) Multi-resolution hash coding of position information; specifically: Step 1: Normalize spatial coordinates; Step 2: Determine the hash-coded multi-resolution levels; Step 3: Hash the vertices of the voxel grid at each resolution level; Step 4: Linearly interpolate the feature vector to match the input coordinates; (2) Spherical harmonic function encoding of direction information: first normalize the direction vector Convert to spherical coordinate direction vector ; Then, the direction information of the sampling point is encoded through the spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding to achieve directional feature embedding; the specific calculation steps are as follows: in, is the order of the spherical harmonics, is the order of the spherical harmonics and l, is the associated Legendre polynomial, is the normalization factor; (3) Image appearance encoding: An additional neural network layer Embedding layer is embedded before the input of the color value prediction module. This layer maps each input image to a low-dimensional appearance encoding vector AppearanceEmbeddings.

4. The method according to claim 3, characterized in that In steps three and four, the rendered image is converted into a high-contrast feature edge map or a target area map, which is compared with the true value of the high-contrast feature edge map or the target area map, and the loss between the generated image and the true value is calculated; the loss function use L 2-norm loss, calculated as: (13) Where, is the volume rendering loss, is the super-resolution loss, Enhance loss for edges; 、 、 is the weighting coefficient of different loss function components; is the set of all sampled pixels from the model training image dataset, is the set of all sampled pixels from the high-resolution image dataset; is the color value of the known pixel on the low-resolution image, The pixel color value of the model body rendering image, is the color value of the known pixel on the high-resolution image, Generate pixel color values ​​of the image for the super-resolution module, The color value of the pixel in the focus area of ​​the model training image, The color value of the pixel in the focus area of ​​the model body rendering image.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images