Building group multi-scale high-efficiency high-precision digital twinning multi-resolution neural radiation field method

Abstract

The invention provides a building group multi-scale high-efficiency high-precision digital twinning multi-resolution neural radiation field method. The method comprises the following steps of: 1, acquiring a multi-view apparent image of a building group and generating a multi-resolution data set; 2, establishing a multi-resolution neural network model for building group scene rendering color and volume density estimation; 3, training a building group scene rendering and three-dimensional reconstruction multi-resolution neural network model by adopting a super-resolution enhancement body rendering technology; and 4, performing multi-scale building group scene rendering and new view synthesis by adopting the trained multi-resolution neural network model. According to the method, any-view-angle image rendering, high-efficiency and high-precision three-dimensional reconstruction and digital twinning of a multi-scale large-scale building group scene are realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, three-dimensional reconstruction, scene rendering, digital twins, and intelligent operation and maintenance technology of building structures, and in particular to a multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes. The method is dedicated to solving the problem of multi-resolution, high-precision three-dimensional modeling and detail rendering in scenes adapted to multi-scale building complexes, and can be widely used in the fields of building complex visualization and virtual reality, structural health monitoring and disaster prevention and mitigation, architectural design and planning, and intelligent construction. Background Art

[0002] 3D reconstruction technology aims to restore the 3D structure of a scene from multi-view 2D images. Traditional 3D reconstruction technology is based on geometric methods, including multi-view stereo vision (MVS) and laser scanning, and achieves high-precision 3D reconstruction through explicit modeling. There are the following challenges in using the above traditional methods to map the apparent damage and update the 3D model of the building complex structure in real time: (1) the lack of multi-scale processing and automated editing level for building complex scenes, (2) it is highly restricted by the accuracy of image feature matching, and has poor performance in target areas with sparse or repeated textures, (3) it relies heavily on manual operation and professional modeling software, with insufficient intelligence, low efficiency and capabilities, and (4) it requires high-performance perception equipment (such as 3D laser point cloud scanners, etc.).

[0003] In recent years, deep learning-driven implicit modeling methods such as Neural Radiance Field (NeRF) have achieved high-quality new-perspective image generation by directly fitting the radiation field function of a three-dimensional scene through a neural network. The advantages of the NeRF method in high-quality three-dimensional reconstruction and perspective-independent real-time rendering provide potential possibilities for three-dimensional reconstruction, scene rendering, and detail presentation in complex scenes of building complexes, and bring new ideas for the realization of multi-scale three-dimensional reconstruction of building complexes, structural damage mapping, and real-time digital twins. However, the NeRF method still has problems such as high computational complexity, insufficient expression of details, and poor adaptability to large-scale scenes. However, these inherent problems become particularly prominent when it comes to the three-dimensional reconstruction of large-scale building complexes, posing new challenges for the realization of efficient and high-precision three-dimensional reconstruction and digital twin modeling of multi-scale building complexes:

[0004] (1) Building complex visual data (multi-view images or videos) contain multi-scale information, while traditional NeRF methods often only have good performance for a single small scale;

[0005] (2) The digital twin of a building complex has a multi-resolution modeling requirement. That is, when you don’t care about the details, you only need to present the coarse-scale modeling results, and when you care about the details, you need to present the fine-scale modeling results. However, the traditional NeRF method can only model at a single scale and cannot adapt to the automatic adjustment requirements of multiple resolutions.

[0006] (3) If the traditional NeRF method is directly used to deal with the problem of modeling large-scale, detailed building complex scenes, there are problems such as poor detail feature capture capability (low precision) of a single neural network implicit proxy model and long rendering time (low efficiency). Summary of the invention

[0007] The purpose of the present invention is to solve the problems in the prior art and propose a multi-resolution neural radiation field method for multi-scale, efficient and high-precision digital twins of building complexes. It can realize arbitrary perspective image rendering, high-efficiency and high-precision three-dimensional reconstruction and digital twins of multi-scale large-scale building complex scenes.

[0008] The present invention is realized by the following technical scheme. The present invention proposes a multi-resolution neural radiation field method for multi-scale, efficient and high-precision digital twins of building complexes. The method comprises the following steps:

[0009] Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets;

[0010] Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation;

[0011] Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction;

[0012] Step 4: Use the trained multi-resolution neural network model to render multi-scale building scene and synthesize new views.

[0013] Furthermore, the step 1 specifically includes the following steps:

[0014] Step 1: Collect multi-view apparent image data of the building complex, and convert the single-resolution image of the target scene into a multi-resolution image through the image super-resolution generation model. For the collected image data, first convert the original resolution image into a high-resolution image through the image super-resolution model SR, that is, represents the high-resolution image after super-resolution, SR is the image super-resolution pre-training generation model adopted; then Downsampling divides image data into different resolutions Right now:

[0015]

[0016] Among them, Subsample means downsampling the image, r means the resolution level of the image, and k means the highest resolution image. The resolution level of

[0017] Step 1 and 2: Estimate the parameters of the image data to obtain the internal and external parameters of the camera, and calculate the transformation matrix from the camera coordinates to the world coordinates and the relative position and posture of the camera;

[0018] Conversion from pixel coordinate system to camera coordinate system:

[0019]

[0020] In the formula, is the camera coordinate system, O uv is the image pixel coordinate system, K is the camera’s intrinsic parameter matrix, and f x , f y are the focal lengths of the camera in the x and y directions, respectively, (c x , c y ) is the principal point coordinate of the camera;

[0021] Coordinate transformation from camera coordinate system to world coordinate system:

[0022]

[0023] In the formula, is the world coordinate system, c2w is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix R i and the translation vector T i .

[0024] Furthermore, the step 2 specifically includes the following steps:

[0025] Step 21: Perform discrete light segment sampling of key feature target area markers;

[0026] Step 22: Taking images of different resolutions as input, embedding the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding and the image appearance coding, to form a coding vector containing information of different scales;

[0027] Step 23: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene space of the building complex; k The spatial coordinates P(x, y, z), the normalized direction vector d and the image appearance embedding are taken as input, and a multi-resolution neural network model is established to output the color value c and volume density value σ of each sampling point.

[0028] Furthermore, the step 21 specifically includes:

[0029] (1) Camera ray generation based on normalized direction vector;

[0030] (2) Marking of key feature target areas;

[0031] (3) importance sampling of discrete rays in a general area;

[0032] (4) Optimization of discrete light segment sampling for key feature target areas.

[0033] Furthermore, the step 22 specifically includes:

[0034] (1) Multi-resolution hash coding of location information; specifically:

[0035] Step 1: Normalize spatial coordinates;

[0036] Step 2: Determine the multi-resolution level of the hash code;

[0037] Step 3: Hash the vertices of the voxel grid at L resolution levels;

[0038] Step 4: Linearly interpolate the feature vector to match the input coordinates;

[0039] (2) Spherical harmonic encoding of directional information: First, normalize the directional vector Convert to spherical coordinate direction vector Then, the direction information of the sampling point is encoded through the spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding to achieve directional feature embedding; the specific calculation steps are as follows:

[0040]

[0041] Where l is the order of the spherical harmonics, m is the sequence of the spherical harmonics and -l<m<1, is the associated Legendre polynomial, is the normalization factor;

[0042] (3) Image appearance encoding: An additional neural network layer Embeddinglayer is embedded before the input of the color value prediction module. This layer maps each input image to a low-dimensional appearance encoding vector AppearanceEmbeddings.

[0043] Furthermore, the step three specifically includes:

[0044] Step 31: Generate pixel color prediction values ​​of multi-view images of building complexes using volume rendering technology;

[0045] Step 32: Generate a high-resolution enhanced image through a super-resolution module;

[0046] Step 33: extract feature edges or target areas through edge detection or semantic segmentation algorithm, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume rendered feature edge map or target area map;

[0047] Step 3 and 4: Calculate the weighted sum of the L2 norm loss and train the color and volume density estimation neural network for building scene rendering and 3D reconstruction in a supervised learning manner.

[0048] Furthermore, in steps three and four, the rendered image is converted into a high-contrast feature edge map or a target area map, which is compared with the true value of the high-contrast feature edge map or the target area map, and the loss between the generated image and the true value is calculated; the loss function The L2 norm loss is used and the calculation method is:

[0049]

[0050] In the formula, is the volume rendering loss, is the super-resolution loss, is the edge enhancement loss; ω1, ω2, ω3 are the weighted coefficients of different loss function components; Ω1 is the set of all sampled pixel points from the model training image dataset, and Ω2 is the set of all sampled pixel points from the high-resolution image dataset; C low (r) is the color value of the known pixel on the low-resolution image, is the pixel color value of the model body rendering image, C high (r) is the color value of the known pixel on the high-resolution image, The pixel color value of the image generated by the super-resolution module, C edge (r) is the color value of the pixel in the focus area of ​​the model training image, The color value of the pixel in the focus area of ​​the model body rendering image.

[0051] Furthermore, the step 4 specifically includes:

[0052] (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and perform sampling on the rays;

[0053] (2) The spatial coordinates P (x, y, z) of the sampling point and the direction vector d are used as input to the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value c (s k ) and volume density value σ;

[0054] (3) Combining volume rendering and super-resolution technology, the color values ​​of image pixels at different resolutions can be predicted to achieve multi-scale new perspective image generation.

[0055] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-resolution neural radiation field method of multi-scale, efficient and high-precision digital twins of the building complex are implemented.

[0056] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The light sampling module is optimized, combining segmented sampling and enhanced sampling of key areas of interest, dynamically adjusting the sampling density, and achieving more detailed high-resolution rendering modeling for areas with rich details such as edges, corners, and textures of building complexes, thereby achieving high-precision improvement;

[0059] (2) Through the fusion of multi-resolution hash coding spatial features, global structures and local details are captured simultaneously at different spatial scales, the representation ability and rendering accuracy of multi-scale 3D reconstruction scenes of building complexes are enhanced, and multi-resolution improvement is achieved;

[0060] (3) Introducing a super-resolution module to significantly improve image clarity and accuracy, enhance the model's detailed representation of textures, edges, and fine structures, avoid the high cost of direct high-resolution rendering modeling, and achieve high accuracy and efficiency improvements;

[0061] (4) Design a small multi-layer perceptron (MLP) architecture for hash coding, volume density prediction, and color value prediction modules. By using hash coding, the computational efficiency is significantly improved and memory consumption is reduced. At the same time, the small MLP also reduces model parameters and computational complexity, thereby accelerating the training and reasoning process, improving the efficiency of real-time 3D modeling and scene rendering, and achieving high efficiency improvement.

[0062] (5) It is applicable to multi-view images or video visual data of building complexes containing multi-scale information, overcoming the limitation of traditional NeRF methods that they are often only modeled at small scales;

[0063] (6) It realizes multi-resolution modeling of digital twins of building complexes. That is, when details are not important, only coarse-scale modeling results need to be presented, and when details are important, fine-scale modeling results are presented. This breaks through the limitation of the traditional NeRF method that it can only model at a single scale and cannot adapt to multi-resolution automatic adjustment.

[0064] (7) By designing efficient computing modules for hash coding, volume density prediction, and color value prediction, the problems of large size and long rendering time of a single neural network implicit proxy model existing in directly using the traditional NeRF method to model large-scale building complexes are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0066] Figure 1 It is a flow chart of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes.

[0067] Figure 2 It is a schematic diagram of the architecture of the super-resolution generation model for multi-view apparent images of building complexes.

[0068] Figure 3 It is a schematic diagram of the initial segmented sampling process of discrete rays in a general area.

[0069] Figure 4 Schematic diagram of the discrete ray importance sampling process for a general area.

[0070] Figure 5 It is a schematic diagram of the discrete light enhancement sampling process considering the marking of the target area of ​​key features of the building complex.

[0071] Figure 6 It is a schematic diagram of a multi-resolution hash encoder for position information.

[0072] Figure 7 Schematic diagram of the architecture of a multi-resolution neural radiance field model for color and volume density estimation of three-dimensional scenes of building complexes.

[0073] Figure 8 It is a schematic diagram of the architecture of the two-dimensional image super-resolution enhancement module of the building complex neural radiation field model.

[0074] Fig. 9 It is a schematic diagram of scene rendering and new view synthesis based on the pre-trained 3D reconstruction model of the building complex. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Combination Figure 1-Figure 9 The present invention proposes a multi-resolution neural radiation field method for multi-scale, efficient and high-precision digital twins of building complexes, and the method comprises the following steps:

[0077] Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets;

[0078] The step 1 specifically includes the following steps:

[0079] Step 1: Collect multi-view apparent image data of the building complex, and transform the single-resolution image of the target scene into a multi-resolution image through image super-resolution generation model processing; use a high-resolution camera to collect 360° full-view panoramic image data of the target scene building complex (denoted as I i , where i is the image number), to obtain comprehensive surface information of the building complex. For the collected image data, the original resolution image is first converted into a high-resolution image through the image super-resolution model (Super-Resolution, referred to as SR), that is, represents the high-resolution image after super-resolution, SR is the image super-resolution pre-training generation model adopted; then Downsampling divides image data into different resolutions Right now:

[0080]

[0081] Among them, Subsample means downsampling the image, r means the resolution level of the image, and k means the highest resolution image. The resolution level of

[0082] Select specific high-resolution image data levels based on application requirements As model training input (m represents the selected r value, ranging from 1, 2, ..., k), and a specific high-resolution image and specific low-resolution image data As a benchmark dataset for calculating model loss, the architectural scene representations at different resolution scales are obtained through image data super-resolution and downsampling. The lower-resolution images focus on the overall representation of the architectural scene, while the high-resolution images ensure that the detailed features of the building (such as building edges and building surface textures) are fully captured and preserved.

[0083] Step 1 and 2: Estimate the parameters of the image data to obtain the internal and external parameters of the camera, and calculate the transformation matrix from the camera coordinates to the world coordinates and the relative position and posture of the camera;

[0084] The high-resolution image set will be selected As data input, use image processing tools (such as COLMAP) to estimate the parameters of the collected building scene image data, obtain the camera intrinsic parameters (camera intrinsic parameter matrix K), camera extrinsic parameters (rotation matrix R and translation vector T) and the sparse reconstruction results of the scene. Then, combine the camera intrinsic and extrinsic parameters with the original 2D image information to convert the 2D coordinates of the image information into 3D coordinates in the world coordinate system. The specific steps are as follows:

[0085] Conversion from pixel coordinate system to camera coordinate system:

[0086]

[0087] In the formula, is the camera coordinate system, O uv is the image pixel coordinate system, K is the camera’s intrinsic parameter matrix, and f x , f y are the focal lengths of the camera in the x and y directions, respectively, (c x , c y ) are the principal point coordinates of the camera (the center of the image);

[0088] Coordinate transformation from camera coordinate system to world coordinate system:

[0089]

[0090] In the formula, is the world coordinate system, c2w is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix R i and the translation vector T i .

[0091] Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation;

[0092] The step 2 specifically includes the following steps:

[0093] Step 21: Perform discrete light segment sampling of key feature target area markers; specifically including:

[0094] (1) Camera ray generation based on normalized direction vector:

[0095] Subtract the optical center coordinate (c x , c y ), get the absolute direction vector (ic) in homogeneous coordinates x , jc y , f), normalize the focal length f as the z-axis coordinate division to obtain the normalized direction vector under homogeneous coordinates Based on the c2w transformation matrix obtained in steps one and two, the camera center coordinates and the normalized direction vector are transformed into the world coordinate system to form a light ray r(t)=o+td, where o represents the optical center, t represents the distance along the direction of the light, and d represents the normalized direction vector.

[0096] (2) Marking of key feature target areas:

[0097] The edge feature map or target area map is extracted from the original high-resolution image through edge detection or semantic segmentation algorithm, and the image edge or target area is marked as high value and the background area as low value to obtain a high-contrast enhanced sampling image (denoted as ). The edge detection algorithm is used to enhance the high-contrast areas in the sampled image, which represent the edges, corners, textures and other detail-rich areas of the building complex; the semantic segmentation algorithm is used to extract the pixel set where the main buildings in the scene are located, which is the main target area for 3D reconstruction of the building. These marked targets will be processed in the subsequent discrete light segmentation sampling process.

[0098] (3) Importance sampling of discrete rays in a general area:

[0099] The camera ray generation and discrete ray sampling process based on the normalized direction vector is as follows Figure 3 As shown. Discretize along the direction of the light to obtain a finite number of sampling points, and use the importance sampler to sample a set of orthogonal points between the near boundary near and the far boundary far along the light. Among them, o is the optical center, c is the corresponding coordinate of the optical center on the imaging plane, s1~s n is the sampling point on the ray r(s)=o+td. The discrete ray importance sampler for a general area is as follows Figure 4 As shown, the specific sampling steps are:

[0100] Step 1: Preliminary piecewise sampling (Piecewise Sampling).

[0101] At a fixed distance near the camera [t n , t m], a uniform sampling strategy is used to generate sample points s with consistent intervals. i ; The longest distance t beyond uniform sampling m Finally, the linear sampling strategy (LinearSampling) is used, and the sample step size increases linearly with the increase of distance, and the sampling is continued until the farthest distance from the camera t f Such a combined sampling strategy allows for efficient sampling of distant objects while still maintaining a dense set of samples for nearby objects. The sampling process can be expressed as:

[0102] s i ~Uniform(t n , t m ), i=1,2,...,N1 (4)

[0103] s j =t m +(j·Δs),t f =max(s j ), j = 1, 2, ..., N2 (5)

[0104] Among them, s i represents the position of the ith sample near the camera, and Uniform represents the position of the ith sample near the camera. n , t m ] uniform sampling within s j represents the position of the jth sample at the far camera end to the near camera end, and Δs is the step size that increases as j increases.

[0105] Step 2: Proposal Network Sampling.

[0106] The main goal of the proposal network sampling is to integrate the sample position into the area of ​​the scene that contributes most to the final rendering, usually the first surface intersection. First, a small fusion multi-layer perceptron (MLP) and hash coding are used to represent the density function of the scene:

[0107] σ(s)=DensityFild(hash(s)) (6)

[0108] According to the density σ(s) predicted by the proposal network, a probability density function (PDF) sampler is constructed and a new set of samples is generated. The total number of these new samples (N = N1 + N2) remains unchanged and is located at the position existing in the predicted scene:

[0109] s~PDF(σ(s)) (7)

[0110] Where s′ is the position of the resampled sample point, and PDF(σ(s)) is the probability density function based on the predicted density σ(s). At the same time, the proposed network sampler can work in chain with multiple density functions to further integrate the sampling. The present invention uses two density functions, and more than two may lead to diminishing returns.

[0111] (4) Optimization of discrete light segment sampling for key feature target areas

[0112] In addition to sampling enhancement on the same ray, since different rays representing the scene contain inconsistent target information, an optimization strategy is also used for sampling different rays. Figure 5 As shown. For (2) the high contrast enhanced sampling image obtained in the key feature target area marking The pixel values ​​are weighted to obtain the sampling weight map of different rays. When sampling each ray, the total number of sampling points N on the ray is multiplied by the corresponding weight e in the ray sampling weight map. j , get the total number of new light sampling points N′=e j N. If the light intersects the key area, the weight value e j If it is greater than 1, the sampling density of the ray is increased, and the additional sample points are evenly interpolated in the original sample point distribution; if the ray passes through a non-key area, the weight value e j Equal to 1, that is, the original sampling density of the light is used.

[0113] Step 22: Take images of different resolutions as input, embed the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding and the image appearance coding, and form a coding vector containing information of different scales; the coding process specifically includes:

[0114] (1) Multi-resolution hash coding of position information

[0115] The spatial coordinates of the sampling points are encoded through a hash encoder to achieve position feature embedding and obtain the image sampling point position feature vector based on multi-resolution hash coding. Taking the multi-resolution hash encoder of position information in two-dimensional space as an example, Figure 6 As shown, the specific steps are as follows.

[0116] Step 1: Normalize spatial coordinates

[0117] First, normalize the dimensions of the input spatial coordinates p = (x, y, z) to the range [0, 1]:

[0118]

[0119] Where min(x) and max(x) are the minimum and maximum values ​​of the original space coordinates, respectively.

[0120] Step 2: Determine the hash-encoded multi-resolution levels

[0121] The sampling space is divided into different detail levels L according to different resolutions. i Divided into several uniform spatial grids, each resolution level L i There are k hash coding levels corresponding to a different hash encoder using a special hash function. Each level L i The number of grid points N i and the common eigenvector dimension F. The number of grid points N i By level L i Increase and decrease to capture feature information of different scales. It can be set as a geometric sequence, such as N i =N min b i-1 , where N min is the minimum number of grid points, and b is the scaling factor.

[0122] Step 3: Hash the vertices of the voxel grid at L resolution levels.

[0123] Use a special hash function to convert the three-dimensional grid coordinates p grid Mapping to hash values ​​h at different resolution levels i , and then look up the hash code vector from the pre-set hash table:

[0124]

[0125] Among them, v i is the hash code vector (whose dimension is F) mapped by the grid space coordinates in the hash table, Hash functions for implementing simple integer mappings (such as linear congruential generators).

[0126] Step 4: Linearly interpolate the feature vectors to match the input coordinates.

[0127] For each resolution level L i , the normalized spatial coordinates p′ of any point are mapped to the grid index, grid , in Indicates rounding down, grid index I grid Represents the normalized space coordinate P′ in the Lth i The grid position to which the level belongs. By grid index I gridDetermine the nearest voxel grid around the spatial coordinates, use the hash table to obtain the encoding vector for weighted interpolation, and concatenate the feature encodings of all resolution levels to obtain the position encoding of any sampling point. The specific calculation steps are as follows:

[0128]

[0129] in, is the hash code vector of the nearest voxel grid point around the spatial coordinate p′, w k is the interpolation weight of each voxel grid, v′ i is the spatial coordinate p′ at the resolution level L i The hash code vector under φ(p) represents the position code vector of any sampling point p′ in the space obtained by concatenating the feature codes at all resolution levels.

[0130] (2) Spherical harmonic encoding of directional information

[0131] First, normalize the direction vector Convert to spherical coordinate direction vector Then, the direction information of the sampling point is encoded through the spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding, realizing the embedding of direction features. The specific calculation steps are as follows:

[0132]

[0133] Where l is the order of the spherical harmonics, m is the sequence of the spherical harmonics and -l<m<1, is the associated Legendre polynomial, is the normalization factor.

[0134] (3) Image appearance coding

[0135] An additional layer of neural network (Embedding layer) is embedded before the input of the color value prediction module, which maps each input image to a low-dimensional appearance encoding vector (Appearance Embeddings). By embedding the learnable appearance encoding vector, the model optimizes an appearance encoding vector for each training image during training, which can capture the specific appearance features of the image, helping the model to correct exposure differences under different shooting conditions during rendering, thereby generating more consistent and realistic view synthesis results.

[0136] Step 23: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene space of the building complex; kThe spatial coordinates P(x, y, z), the normalized direction vector d and the image appearance embedding are used as input to establish a multi-resolution neural network model, which outputs the color value c and volume density value σ of each sampling point. The model architecture is as follows Figure 7 As shown, it includes a position information embedding module, a direction information embedding module, an appearance coding embedding module, a volume density prediction module, and a color value prediction module.

[0137] (1) Position information embedding module: that is, the position information multi-resolution hash coding module in step 22.

[0138] (2) Direction information embedding module: that is, the spherical harmonic function encoding module of the direction information in step 22.

[0139] (3) Appearance coding embedding module: the embedding network module of the image appearance coding in step 22

[0140] (4) Volume density prediction module: It uses a small multi-layer perceptron (MLP), i.e., a multi-layer neural network structure, inputs a feature vector φ(p) containing hash codes of the spatial positions of sampling points at different levels, and outputs a spatial feature vector and a volume density prediction value σ corresponding to the spatial coordinates.

[0141] (5) Color value prediction module: It uses a small multi-layer perceptron MLP, i.e., a multi-layer neural network structure. Its input includes: the sampling point direction feature vector SH(d), the spatial feature vector output by the volume density prediction module, and the appearance embedding encoding vector of the original image. It outputs the color scalar prediction value c(C corresponding to the spatial coordinates). k ).

[0142] Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction;

[0143] The step three specifically includes:

[0144] Step 31: Generate pixel color prediction values ​​of multi-view images of building complexes using volume rendering technology;

[0145] Step 3: Use volume rendering technology to generate pixel color prediction values ​​of the multi-view image of the building complex, and generate a low-resolution image through discretized volume rendering technology. The specific steps are:

[0146] Extract N′ sample points t from the light along the light path according to the sampling strategy in step 21 i , weighted sum the color contributions of the selected sampling points to generate the predicted pixel color value. The weighted sum calculation method for volume rendering discretization is:

[0147]

[0148] Where, t i represents the t value corresponding to the i-th sampling point after discretization on the light, δ i Represents the distance between adjacent sampling points, T i is the transparency function from the viewpoint to the i-th sampling point, α i is the predicted value of the volume density at the i-th sampling point, c i is the color prediction value of the i-th sampling point, Represents the color prediction value of the image pixel after discretization volume rendering.

[0149] Step 32: Generate a high-resolution enhanced image through a super-resolution module;

[0150] Step 32 converts the low-resolution image generated in step 31 into a high-resolution enhanced image through the super-resolution module, reducing the computational overhead and training time of multi-resolution modeling. The module architecture is as follows: Figure 8 As shown, specifically including:

[0151] (1) The super-resolution module is used as an image enhancement module in the form of a pre-trained super-resolution model, and does not participate in the training and parameter optimization of the neural radiation field model, further reducing the computational cost.

[0152] (2) The training data used by the super-resolution model include downsampled images of the original high-resolution images and low-resolution images generated by volume rendering. The super-resolution module is used to enhance the low-resolution rendering results and super-resolve the images to the same resolution level as the highest-resolution images.

[0153] (3) In addition, some parameters of the super-resolution model can be fine-tuned during the radiation field model training process. Although the amount of calculation increases slightly, the fine-tuned super-resolution model can make targeted optimizations for the image information of the building complex target scene and obtain better modeling effects.

[0154] Step 33: extract feature edges or target areas through edge detection or semantic segmentation algorithm, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume rendered feature edge map or target area map;

[0155] Step 3 and 4: Calculate the weighted sum of the L2 norm loss and train the color and volume density estimation neural network for building scene rendering and 3D reconstruction in a supervised learning manner.

[0156] Steps 3 and 4 calculate the model training loss to test the rendering effect of the model on the key areas of the building scene and the model generation effect of the building scene views of different resolution scales. Specifically include:

[0157] (1) Convert the rendered image into a high-contrast feature edge map or target area map, compare it with the true value of the high-contrast feature edge map or target area map obtained in step 2.1, and calculate the loss between the generated image and the true value. Loss function The L2 norm loss is used and the calculation method is:

[0158]

[0159] In the formula, is the volume rendering loss, is the super-resolution loss, is the edge enhancement loss; ω1, ω2, ω3 are the weighted coefficients of different loss function components; Ω1 is the set of all sampled pixel points from the model training image dataset, and Ω2 is the set of all sampled pixel points from the high-resolution image dataset; C low (r) is the color value of the known pixel on the low-resolution image, is the pixel color value of the model body rendering image, C high (r) is the color value of the known pixel on the high-resolution image, The pixel color value of the image generated by the super-resolution module, C edge (r) is the color value of the pixel in the focus area of ​​the model training image, The color value of the pixels in the focus area of ​​the model volume rendering image.

[0160] (2) The neural network model is trained in a supervised learning manner, using a gradient descent-based optimization algorithm (such as Adam, RAdam, SGD, etc.) to ensure that the color prediction value of the pixel in the final super-resolution image is as close as possible to the actual color value of the corresponding pixel in the original high-resolution building scene image. At the same time, the loss between the volume rendering image and the real image and the influence of the sampling of the key attention area are also considered.

[0161] Step 4: Use the trained multi-resolution neural network model to render multi-scale building scene and synthesize new views.

[0162] Step 4: Use the trained multi-resolution neural network to render multi-scale building scene and synthesize new views. The specific process is as follows: Fig. 9 As shown, including:

[0163] (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and perform sampling on the rays;

[0164] (2) The spatial coordinates P (x, y, z) of the sampling point and the direction vector d are used as input to the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value c (sk ) and volume density value σ;

[0165] (3) Combining volume rendering and super-resolution technology, the color values ​​of image pixels at different resolutions can be predicted to achieve multi-scale new perspective image generation.

[0166] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-resolution neural radiation field method of multi-scale, efficient and high-precision digital twins of the building complex are implemented.

[0167] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of the building complex.

[0168] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0170] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0171] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0172] The multi-resolution neural radiation field method for multi-scale, efficient and high-precision digital twins of building complexes proposed in the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technicians in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-resolution neural radiation field method for multi-scale, efficient, and high-precision digital twins of building complexes, characterized by: The method comprises the following steps: Step 1: Acquisition of multi-view surface images of building complexes and generation of multi-resolution datasets; Step 2: Establish a multi-resolution neural network model for building scene rendering color and volume density estimation; Step 3: Use super-resolution enhanced volume rendering technology to train a multi-resolution neural network model for building scene rendering and 3D reconstruction; Step 4: Use the trained multi-resolution neural network model to render multi-scale building scene and synthesize new views.

2. The method according to claim 1, characterized in that The step 1 specifically includes the following steps: Step 1: Collect multi-view apparent image data of the building complex, and convert the single-resolution image of the target scene into a multi-resolution image through the image super-resolution generation model. For the collected image data, first convert the original resolution image into a high-resolution image through the image super-resolution model SR, that is, represents the high-resolution image after super-resolution, SR is the image super-resolution pre-training generation model adopted; then Downsampling divides image data into different resolutions Right now: Among them, Subsample means downsampling the image, r means the resolution level of the image, and k means the highest resolution image. The resolution level of Step 1 and 2: Estimate the parameters of the image data to obtain the internal and external parameters of the camera, and calculate the transformation matrix from the camera coordinates to the world coordinates and the relative position and posture of the camera; Conversion from pixel coordinate system to camera coordinate system: In the formula, is the camera coordinate system, O uv is the image pixel coordinate system, K is the camera’s intrinsic parameter matrix, and f x ,f y are the focal lengths of the camera in the x and y directions, respectively, (c x ,c y ) is the principal point coordinate of the camera; Coordinate transformation from camera coordinate system to world coordinate system: In the formula, is the world coordinate system, c2w is the coordinate transformation matrix from the camera coordinate system to the world coordinate system, which includes the rotation matrix R i and the translation vector T i .

3. The method according to claim 2, characterized in that The step 2 specifically includes the following steps: Step 21: Perform discrete light segment sampling of key feature target area markers; Step 22: Taking images of different resolutions as input, embedding the image sampling point position feature vector based on multi-resolution hash coding, the light direction vector based on spherical harmonic coding and the image appearance coding, to form a coding vector containing information of different scales; Step 23: Establish a multi-resolution neural network model to output the color and volume density values ​​of the sampling points in the three-dimensional scene space of the building complex; k The spatial coordinates P(x, y, z), the normalized direction vector d and the image appearance embedding are used as input to establish a multi-resolution neural network model, which outputs the color value c and volume density value σ of each sampling point.

4. The method according to claim 3, characterized in that: The step 21 specifically includes: (1) Camera ray generation based on normalized direction vector; (2) Marking of key feature target areas; (3) importance sampling of discrete rays in a general area; (4) Optimization of discrete light segment sampling for key feature target areas.

5. The method according to claim 4, characterized in that The step 22 specifically includes: (1) Multi-resolution hash coding of location information; specifically: Step 1: Normalize spatial coordinates; Step 2: Determine the multi-resolution level of the hash code; Step 3: Hash the vertices of the voxel grid at L resolution levels; Step 4: Linearly interpolate the feature vector to match the input coordinates; (2) Spherical harmonic encoding of directional information: First, normalize the directional vector Convert to spherical coordinate direction vector Then, the direction information of the sampling point is encoded through the spherical harmonic function encoder to obtain the light direction vector based on spherical harmonic encoding to achieve directional feature embedding; the specific calculation steps are as follows: Where l is the order of the spherical harmonics, m is the order of the spherical harmonics and -l <m<l, is the associated Legendre polynomial, is the normalization factor; (3) Image appearance encoding: An additional neural network layer Embeddinglayer is embedded before the input of the color value prediction module. This layer maps each input image to a low-dimensional appearance encoding vector AppearanceEmbeddings.

6. The method according to claim 5, characterized in that The step three specifically includes: Step 31: Generate pixel color prediction values ​​of multi-view images of building complexes using volume rendering technology; Step 32: Generate a high-resolution enhanced image through a super-resolution module; Step 33: extract feature edges or target areas through edge detection or semantic segmentation algorithm, perform sampling enhancement on the automatically selected feature edges or target areas, and generate a volume rendered feature edge map or target area map; Step 3 and 4: Calculate the weighted sum of the L2 norm loss and train the color and volume density estimation neural network for building scene rendering and 3D reconstruction in a supervised learning manner.

7. The method according to claim 6, characterized in that In steps three and four, the rendered image is converted into a high-contrast feature edge map or a target area map, which is compared with the true value of the high-contrast feature edge map or the target area map, and the loss between the generated image and the true value is calculated; the loss function The L2 norm loss is used and the calculation method is: In the formula, is the volume rendering loss, is the super-resolution loss, is the edge enhancement loss; ω1, ω2, ω3 are the weighted coefficients of different loss function components; Ω1 is the set of all sampled pixel points from the model training image dataset, and Ω2 is the set of all sampled pixel points from the high-resolution image dataset; C low (r) is the color value of the known pixel on the low-resolution image, is the pixel color value of the model body rendering image, C high (r) is the color value of the known pixel on the high-resolution image, The pixel color value of the image generated by the super-resolution module, C edge (r) is the color value of the pixel in the focus area of ​​the model training image, The color value of the pixel in the focus area of ​​the model body rendering image.

8. The method according to claim 7, characterized in that The step 4 specifically includes: (1) Select the camera center, set the number of rays based on the resolution of the image to be rendered, and perform sampling on the rays; (2) The spatial coordinates P (x, y, z) of the sampling point and the direction vector d are used as input to the trained multi-resolution neural network model for building scene rendering and 3D reconstruction, and the color value c (s k ) and volume density value σ; (3) Combining volume rendering and super-resolution technology, the color values ​​of image pixels at different resolutions can be predicted to achieve multi-scale new perspective image generation.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images