A Sparse-View 3D Building Reconstruction Method and Device Based on Generative Sequential Frame Images

Abstract

The present invention belongs to the field of visual three-dimensional reconstruction, and provides a sparse-view building three-dimensional reconstruction method and device based on generated sequential frame images. The present invention generates building images by using a trained LoRA model, inputs the building images into the ComfyUI platform, combines AI video generation technology to generate a target building video, performs video image processing on the target building video to generate a sequential frame image set, generates a pose data set through the SfM algorithm, and inputs the pose data set and the sequential frame image set into the NSVF model to reconstruct and generate a three-dimensional building model. Finally, a loss function is introduced to reduce rendering loss and optimize the structure to obtain the final three-dimensional building model. The present invention combines technical means such as AI video generation technology, OpenCV frame division, and NSVF three-dimensional reconstruction to achieve high-precision three-dimensional model reconstruction of buildings, meeting the needs of digital protection and virtual reproduction of cultural heritage.

Description

Technical Field

[0001] The present invention belongs to the field of visual three-dimensional reconstruction, and particularly relates to a sparse-view building three-dimensional reconstruction method and device based on generative sequence frame images. Background Art

[0002] With the development of three-dimensional reconstruction technology, generating a digital three-dimensional model of a building using image sequence frames has become an important technical means in cultural heritage protection and restoration. Traditional methods mainly rely on technical means such as multi-view stereo vision (MVS), structured light, and lidar to reconstruct the three-dimensional model by capturing the appearance and geometric information of the building. These methods have high technical requirements and costs in image acquisition and data processing, and are relatively sensitive to external conditions such as environmental lighting, weather, and shooting angles.

[0003] In recent years, the NSVF voxel neural field technology has gradually attracted attention as an emerging three-dimensional reconstruction method. NSVF can use machine learning algorithms to generate a high-precision three-dimensional model by inputting multi-view sequence frame images of a building and effectively restore the building details. However, the current NSVF-based reconstruction technology still faces certain challenges in processing complex building details and textures. The inherent defects of the NSVF technology are as follows: First, for environmental dependence, traditional multi-view stereo vision and structured light technologies are significantly affected by environmental lighting and are prone to errors when the external light source changes greatly. When NSVF is training and inferring, stable lighting conditions are still required to obtain high-quality reconstruction results, which limits the flexibility of its application scenarios in practice; Second, for high computational resource requirements, the NSVF technology requires a large amount of computational resources during the reconstruction process, resulting in difficulty in realizing real-time model generation on ordinary computer devices. In addition, generating a large-scale and detailed building model requires several hours or even longer computing time, which cannot meet the requirement of high efficiency; Third, for insufficient model detail restoration, the current NSVF technology still has deficiencies in building detail processing, especially the restoration effect of fine textures, materials, and light and shadow details of the building is not ideal, which cannot meet the requirements of high-precision digital protection and display; Finally, for high data requirements, the model training of NSVF requires a large amount of multi-angle image data, and the image acquisition of the building is limited by factors such as time, weather, and location, and it is difficult to obtain high-quality data with multi-view and full coverage. Summary of the Invention

[0004] The present invention aims at the deficiencies of the prior art, and combines technical means such as AI video generation technology, OpenCV frame division, and NSVF three-dimensional reconstruction to achieve high-precision three-dimensional model reconstruction of buildings and meet the needs of digital protection and virtual reproduction of cultural heritage.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] The present invention provides a sparse-view three-dimensional reconstruction method for buildings based on generative sequence-frame images, comprising the following steps:

[0007] S1. Collect information on a certain type of building in a certain period, generate a structured text of building information features and the building images corresponding to the text, denoted as dataset X1;

[0008] S2. Input dataset X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building-style LoRA model;

[0009] S3. Input a style description prompt into the ComfyUI platform, combine it with the trained building-style LoRA model to generate multiple building images, and select one building image that meets the preset image criteria;

[0010] S4. Input a subject and action description prompt into the ComfyUI platform, and input the building image selected in step S3 into the ComfyUI platform. Combine it with a lens track model and an AI video generation model to generate target building videos on multiple different track planes, and select multiple target building videos on different track planes that meet the preset video criteria;

[0011] S5. Perform video image processing on the multiple target building videos on different track planes selected in step S4 to generate multiple sequence-frame images corresponding to the target building videos on multiple track planes, and integrate the multiple sequence-frame images to form a sequence-frame image set X2;

[0012] S6. Preprocess the sequence-frame image set X2;

[0013] S7. Form a pose dataset Pose1 from the preprocessed sequence-frame image set X2';

[0014] S8. Input the preprocessed sequence-frame image set X2' and the pose dataset Pose1 into the NSVF model, and reconstruct and generate a three-dimensional building model through the NSVF model;

[0015] S9. Train the NSVF model by introducing a loss function to reduce the rendering loss and optimize the structure to obtain the final three-dimensional building model.

[0016] Preferably, in step S1, the structured text of building information features includes: building style features, building structure features, building material features, and building decoration features of a certain type of building in a certain period.

[0017] Preferably, in step S5, the target building videos on multiple different orbital planes selected are subjected to video image processing by using the method of extracting frames with OpenCV.

[0018] Preferably, in step S6, each frame image in the sequence frame image set X2 is preprocessed, and the preprocessing methods include denoising, brightness equalization, and contrast enhancement.

[0019] Preferably, in step S7, the preprocessed sequence frame image set X2' is subjected to camera pose calculation, and a pose data set Pose1 is formed through the SfM algorithm in computer vision.

[0020] Preferably, the steps included in the SfM algorithm are camera projection model, feature matching and initial pose estimation, triangulation, and bundle adjustment.

[0021] Preferably, step S8 specifically includes:

[0022] S801. Divide the three-dimensional space into sparse voxel grids through the NSVF model, and use the sparse voxel grids to represent the structure and position of the three-dimensional building model;

[0023] S802. Input the preprocessed sequence frame image set X2' and the pose data set Pose1 into the NSVF model for feature extraction. The sparse voxel grid contains multiple sparse voxels. In each sparse voxel, use the neural network in the NSVF model to learn the extracted features, and output the color and density of the voxel, so as to learn the complex details and textures on the surface of the three-dimensional building model in the three-dimensional space;

[0024] S803. According to the structure and position of the three-dimensional building model obtained in step S801 and the color and density of the voxel obtained in step S802, generate a three-dimensional building model through voxel sampling and volume rendering reconstruction of the NSVF model.

[0025] Preferably, step S9 specifically includes:

[0026] S901. Obtain the three-dimensional building model in step S8, export the rendered images in the three-dimensional building model to generate a rendered image set, obtain the preprocessed sequence frame image set X2' in step S6, introduce a loss function with the goal of minimizing the difference between the rendered image set and the preprocessed sequence frame image set X2', and train the NSVF model to reduce the rendering loss;

[0027] S902. Fill and optimize the sparse voxel grid by training the NSVF model;

[0028] S903. Perform labeling processing on the materials and structures of the three-dimensional building model, and store them in layers according to the structure to obtain the final three-dimensional building model.

[0029] The present invention also provides a sparse-view building three-dimensional reconstruction device based on generative sequence frame images. This device uses the sparse-view building three-dimensional reconstruction method based on generative sequence frame images as described above, and includes:

[0030] A building information feature structured text generation module, a model training module, a building image generation module, a target building video generation module, a sequence frame image set generation module, a pose data set generation module, and a three-dimensional building model reconstruction module;

[0031] The building information feature structured text generation module is used to collect building information of a certain type in a certain period, generate building information feature structured text and the corresponding building images of the text;

[0032] The model training module is used to train the LoRA model with the building images corresponding to the building information feature structured text generated by the building information feature structured text generation module;

[0033] The building image generation module is used to input scene prompt words to the ComfyUI platform, generate multiple building images in combination with the LoRA model trained by the model training module, and select the building images that meet the preset image standards;

[0034] The target building video generation module is used to input action prompt words to the ComfyUI platform, input the building images selected by the building image generation module to the ComfyUI platform, generate target building videos of multiple different track planes in combination with the lens track class model and the AI video generation class model, and select target building videos of multiple different track planes that meet the preset video standards;

[0035] The sequence frame image set generation module is used to perform video image processing on the target building videos selected by the target building video generation module, generate a sequence frame image set, and preprocess the sequence frame image set;

[0036] The pose data set generation module is used to process the sequence frame image set preprocessed by the sequence frame image set generation module to generate a pose data set;

[0037] The three-dimensional building model reconstruction module is used to reconstruct a three-dimensional building model through the NSVF model with the sequence frame image set preprocessed by the sequence frame image set generation module and the pose data set generated by the pose data set generation module.

[0038] Compared with the prior art, the beneficial effects produced by the present invention are:

[0039] (1) The 3D model generated by the present invention can be applied to multiple scenarios, used in virtual displays and interactive experiences, imported into augmented reality (AR) and virtual reality (VR) systems for digital display and interactive experiences of buildings, and used in the protection and restoration of cultural heritage. It can provide high-precision 3D building data and offer scientific references for building restoration and reconstruction.

[0040] (2) The present invention combines AI video generation, OpenCV image frame extraction, and NSVF 3D reconstruction to innovatively achieve efficient 3D reconstruction of buildings.

[0041] (3) The present invention utilizes the LoRA model and the ComfyUI platform, combined with AI video generation technology, to achieve high-precision 3D model reconstruction of buildings, which can be widely applied to the reproduction of various buildings, reducing on-site shooting costs and avoiding the limitations of multi-factor influences in on-site shooting.

[0042] (4) By preprocessing the sequence of frame images, the present invention ensures the clarity and consistency of each frame image, reduces noise interference, improves image quality, meets the requirements of detail enhancement and dynamic lighting for 3D building models, and enhances the reproduction effects of details, materials, and textures of 3D building models.

[0043] (5) The high-precision 3D building model generated by the present invention can be applied to various scenarios such as cultural heritage protection, building restoration, and virtual display. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is the technical roadmap of the sparse-view building 3D reconstruction method based on generative sequence frame images of the present invention;

[0045] Figure 2 The rendering flowchart of the NSVF model of the present invention;

[0046] Figure 3 Multi-view effect diagrams of the 3D building model generated by using the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0048] Embodiment 1

[0049] Combined with Figures 1 - 3As shown in the figure, an embodiment of the present invention provides a sparse-view building three-dimensional reconstruction method based on generative sequence frame images, including the following steps:

[0050] S1. Collect information of a certain type of building in a certain period, generate a structured text of building information features and the corresponding building images of the text, denoted as dataset X1;

[0051] S2. Input dataset X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building style LoRA model;

[0052] S3. Input style description prompt words into the ComfyUI platform, and generate multiple building images in combination with the trained building style LoRA model, and select a building image that meets the preset image standards. The specific image preset standards can be the pixel value, clarity, etc. of the building image;

[0053] Among them, the style description prompt words refer to a specific description of the image scene of a certain type of building in a certain period, and the description includes but is not limited to the age, type, material, doors and windows, decoration, special structure, building orientation, etc. of the building in the picture;

[0054] S4. Input subject and action description prompt words into the ComfyUI platform, and input the building image selected in step S3 into the ComfyUI platform, and combine the lens track class model and the AI video generation class model. Specifically, the lens track class model can select the control camera operation track model, and the AI video generation class model can select the cognitive video extension model (CogVideoX) to generate target building videos of multiple different track planes, and select multiple target building videos of different track planes that meet the video preset standards. The specific video preset standards can be the resolution, frame rate, etc. of the target building video;

[0055] Among them, the subject and action description prompt words refer to a specific description of the building video effect to be generated, and the description includes but is not limited to the main target of the generated video, the camera movement trajectory, etc.;

[0056] S5. Perform video image processing on the multiple target building videos of different track planes that meet the video preset standards selected in step S4 to generate multiple sequence frame images corresponding to the target building videos of multiple track planes, and integrate the multiple sequence frame images to form a sequence frame image set X2;

[0057] S6. Preprocess the sequence frame image set X2;

[0058] S7. Form a pose dataset Pose1 from the preprocessed sequence frame image set X2';

[0059] S8. Input the pre - processed sequence frame image set X2' and the pose data set Pose1 into the NSVF model, and reconstruct and generate a 3D building model through the NSVF model;

[0060] In addition, building model reconstruction techniques such as the Neural Radiance Field (NeRF) model and point cloud related to 3D reconstruction can also be used to generate a 3D building model;

[0061] S9. Train the NSVF model by introducing a loss function to reduce the rendering loss and optimize the structure to obtain the final 3D building model;

[0062] Specifically, in step S1, the structured text of building information features includes: building style features, building structure features, building material features, building decoration features, etc. of a certain type of building in a certain period.

[0063] Embodiment 2

[0064] Combined with Figures 1 - 3 As shown, the embodiment of the present invention provides a sparse - view building 3D reconstruction method based on generative sequence frame images, including the following steps:

[0065] S1. Collect building information of a certain type of building in a certain period, generate structured text of building information features and building images corresponding to the text, denoted as data set X1;

[0066] S2. Input data set X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building style LoRA model;

[0067] S3. Input style description prompt words into the ComfyUI platform, and generate multiple building images in combination with the trained building style LoRA model, and select a building image that meets the preset image criteria;

[0068] S4. Input subject and action description prompt words into the ComfyUI platform, and input the building image selected in step S3 into the ComfyUI platform. Combine the lens track - like model and the AI video generation - like model to generate target building videos on multiple different track planes, and select target building videos on multiple different track planes that meet the preset video criteria;

[0069] S5. Perform video image processing on the multiple target building videos on different track planes selected in step S4 to generate multiple sequence frame images corresponding to the target building videos on multiple track planes, and integrate the multiple sequence frame images to form a sequence frame image set X2;

[0070] S6. Pre - process the sequence frame image set X2;

[0071] S7. Form the pose data set Pose1 from the pre - processed sequence frame image set X2'.

[0072] S8. Input the pre - processed sequence frame image set X2' and the pose data set Pose1 into the NSVF model, and reconstruct and generate a 3D building model through the NSVF model.

[0073] S9. Train the NSVF model by introducing a loss function, reduce the rendering loss and optimize the structure to obtain the final 3D building model.

[0074] In step S1, the structured text of building information features includes: architectural style features, architectural structure features, building material features, architectural decoration features, etc. of a certain type of building in a certain period.

[0075] On this basis, in this embodiment, in step S5, the target building videos of multiple different orbital planes selected are subjected to video image processing by using the method of extracting frames with OpenCV to ensure that the complete structure of the building is covered.

[0076] The sequence frame images at a specified interval can be obtained by controlling the frame rate (FPS). Specifically:

[0077] Control the number of frames set X2 extracted by controlling the frame rate FPS. The formula is as follows:

[0078] X2 = V S ×FPS, X2 ∈ V N ,

[0079] where X2 is the number of frames set, V S is the video duration, and V N is the N sets of multi - view target building videos.

[0080] Furthermore, in step S6, pre - process each frame image in the sequence frame image set X2, that is, pre - process each frame image in the sequence frame image set X2. The pre - processing methods include denoising, brightness equalization and contrast enhancement. Let the pre - processed sequence frame image set be X2' to ensure the clarity and consistency of each frame image, reduce noise interference and improve the image quality, and ensure the accuracy of subsequent NSVF reconstruction.

[0081] Among them, brightness equalization and contrast enhancement can be achieved through histogram equalization, which is expressed by the formula as follows:

[0082]

[0083] Where I(x,y) represents the pixel value of the original image at (x,y), min(I) and max(I) represent the minimum and maximum pixel values of the image respectively, and 255 means mapping the normalized value to the range of [0,255] to meet the common grayscale image format;

[0084] Furthermore, in step S7, the pre-processed sequence frame image set X2' is subjected to camera pose calculation to form a pose dataset Pose1 using the SfM algorithm in computer vision. The SfM algorithm includes the steps of camera projection model, feature matching and initial pose estimation, triangulation, and bundle adjustment.

[0085] Specifically:

[0086] 1) Camera projection model: Given a 3D point X, let X = (X, Y, Z), and let the corresponding pixel position of this 3D point on the plane be x = (u, v). The relationship between the two is expressed as:

[0087] x=K·[R|t]·X,

[0088] Among them, K is the intrinsic parameter matrix of the camera, f x and f y is the focal length of the camera in the x and y directions, (c x ,c y ) is the position of the principal point (the intersection of the optical axis and the image plane), R is the camera's rotation matrix, which is used to describe the camera's orientation, and t is the camera's translation vector, which is used to describe the camera's position. The camera's pose can be represented by the matrix [R|t], where R is the rotation matrix and t is the translation vector.

[0089] 2) Feature matching and initial pose estimation: By detecting and matching feature points in the sequence frame images, corresponding point pairs can be obtained. When the camera intrinsic parameter matrix K is known, the geometric relationship between the two images can be described by the essential matrix E. The formula is as follows:

[0090] x2Ex1=0,

[0091] Where E is the essential matrix, E = [t] R, which describes the geometric relationship between the two images, [t] is the antisymmetric matrix of the translation vector t, By calculating the essential matrix for the matching pairs of feature points, we can preliminarily estimate the rotation matrix R and translation vector t between the two images, which is the initial camera pose.

[0092] 3) Triangulation: Once the two initial camera poses [R1|t1] and [R2|t2] are obtained, the triangulation method can be used to recover the 3D point positions. Assuming there are two camera poses [R1|t1] and [R2|t2], and the pixel coordinates of the 3D point X are x1 and x2, then the coordinates of the 3D point can be calculated through the following equations:

[0093] x1 = K·[R1|t1]·X,

[0094] x2 = K·[R2|t2]·X,

[0095] Combining the above two equations, a set of linear equations can be obtained, and the coordinate values of the 3D point X are solved by the least squares method. The set of 3D points X is the pose dataset Pose1;

[0096] 4) Bundle adjustment: Bundle adjustment is a key step in optimizing camera poses and 3D point positions in SfM. The specific formula is as follows:

[0097]

[0098] where Q is the number of cameras, M is the number of 3D points, x ij is the pixel position of the jth 3D point observed by the ith camera, and π(K i , R i , t i , X j ) is the reprojection position calculated using the camera intrinsic matrix K i , the pose [R i |t i and the 3D point position X j .

[0099] Example 3

[0100] Combined with Figures 1 - 3 shown, the embodiment of the present invention provides a sparse-view 3D reconstruction method for buildings based on generative sequence frame images, including the following steps:

[0101] S1. Collect building information of a certain type in a certain period, generate structured text of building information features and corresponding building images, denoted as dataset X1;

[0102] S2. Input dataset X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building style LoRA model;

[0103] S3. Input style description prompt words into the ComfyUI platform, generate multiple building images in combination with the trained building style LoRA model, and select a building image that meets the preset image criteria;

[0104] S4. Input the main body and action description prompts into the ComfyUI platform, and input the building images selected in step S3 into the ComfyUI platform. Combine the lens track class model and the AI video generation class model to generate target building videos with multiple different track planes, and select target building videos with multiple different track planes that meet the video preset standards.

[0105] S5. Perform video image processing on the multiple target building videos with different track planes selected in step S4 to generate multiple sequence frame images corresponding to the target building videos with multiple track planes, and integrate the multiple sequence frame images to form a sequence frame image set X2.

[0106] S6. Preprocess the sequence frame image set X2.

[0107] S7. Form a pose data set Pose1 from the preprocessed sequence frame image set X2'.

[0108] S8. Input the preprocessed sequence frame image set X2' and the pose data set Pose1 into the NSVF model, and reconstruct and generate a 3D building model through the NSVF model.

[0109] S9. Train the NSVF model by introducing a loss function to reduce the rendering loss and optimize the structure to obtain the final 3D building model.

[0110] In step S1, the structured text of the building information features includes: the building style features, building structure features, building material features, building decoration features, etc. of a certain type of building in a certain period.

[0111] On this basis, in this embodiment, step S8 specifically includes:

[0112] S801. Divide the 3D space into sparse voxel grids through the NSVF model, and use the sparse voxel grids to represent the structure and position of the 3D building model.

[0113] Specifically, assume that the coordinate range of the 3D space is (x, y, z), and the size of the sparse voxel grid is d. Then the position of each voxel V(i, j, k) can be defined as:

[0114] V(i, j, k) = (x0 + i·d, y0 + j·d, z0 + k·d),

[0115] where (i, j, k) is the index of the voxel, representing the position in the 3D sparse voxel grid, x0, y0, z0 are the starting coordinates of the space, and d is the side length (resolution) of each voxel, which is used to control the refinement degree of the voxel.

[0116] To optimize the computational efficiency, the NSVF model only retains the voxels containing the surface information of the actual object. Usually, the occupancy probability p of the voxel can be used to determine whether to retain the voxel:

[0117]

[0118] This formula is the occupancy state function, which determines whether the voxel v(i,j,k) is occupied. Among them, p(v(i,j,k)) and occupancy(v(i,j,k)) represent the occupancy probability of the voxel v(i,j,k), which is a binary function: a value of 1 means the voxel is occupied, and a value of 0 means the voxel is empty; fV(i,j,k)contains means if the voxel is occupied; otherwise means the voxel is empty and not occupied;

[0119] S802. Input the preprocessed sequence frame image set X2' and the pose data set Pose1 into the NSVF model for feature extraction. The sparse voxel grid contains multiple sparse voxels. In each sparse voxel, use the neural network in the NSVF model to learn the extracted features, and output the color and density of the voxel, so as to learn the complex details and textures on the surface of the 3D building model in the 3D space;

[0120] S803. According to the structure and position of the 3D building model obtained in step S801 and the color and density of the voxels obtained in step S802, generate a 3D building model through voxel sampling and volume rendering reconstruction of the NSVF model. During the volume rendering process, the model performs voxel sampling and volume rendering of the 3D building model through ray casting according to the obtained color and density of the voxels to reconstruct a complete 3D model;

[0121] Specifically, first perform voxel sampling in the ray direction. Assume that a ray r(q) = a + q·e is emitted from a certain pixel position of the camera,

[0122] where a is the starting point of the ray (the position of the camera), e is the direction of the ray, and q is the step length on the ray, which is used to control the interval of the sampling points;

[0123] NSVF uses the density field σ(X) and the color field c i to generate the color of each pixel. The volume rendering formulas for density and color are as follows:

[0124]

[0125] where C(r) represents the pixel color along the ray r(q), and this color is calculated by weighted accumulation of all volume elements (usually density and color) along the ray; Ti represents the transmittance of light at the sampling point i. The transmittance represents the probability that light is not completely absorbed when propagating from the starting point to point i; α i represents the weight of the i-th sampling point, indicating the contribution intensity of this point; ci represents the color of the i-th sampling point, usually generated by a neural network or a certain color field function, and this value is the local color value of each point;

[0126]

[0127] α i = 1 - exp(-σ i Δ i ),

[0128] where exp is the exponential function, used to represent the exponential attenuation of light with distance in the medium. In volume rendering, the attenuation of light usually follows the exponential model; σ i , σj represent the volume density, usually representing the absorption or scattering coefficient at the voxel, measuring the intensity of light blocked at this point; Δ i , Δ j represent the light step size, that is, the distance that the light travels through in the i-th layer and the j-th layer.

[0129] Example 4

[0130] Combined with Figures 1 - 3 shown, the embodiment of the present invention provides a sparse view building three-dimensional reconstruction method based on generative sequence frame images, including the following steps:

[0131] S1. Collect information of a certain type of building in a certain period, generate structured text of building information features and building images corresponding to the text, denoted as dataset X1;

[0132] S2. Input dataset X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building style LoRA model;

[0133] S3. Input style description prompt words into the ComfyUI platform, and combine with the trained building style LoRA model to generate multiple building images, and select a building image that meets the preset image criteria;

[0134] S4. Input subject and action description prompt words into the ComfyUI platform, and input the building image selected in step S3 into the ComfyUI platform. Combine with the lens track class model and the AI video generation class model to generate target building videos on multiple different track planes, and select target building videos on multiple different track planes that meet the preset video criteria;

[0135] S5. Perform video image processing on the target building videos of multiple different orbital planes that meet the video preset criteria selected in step S4 to generate multiple sets of sequence frame images corresponding to the target building videos of multiple orbital planes, and integrate the multiple sets of sequence frame images to form a sequence frame image set X2;

[0136] S6. Preprocess the sequence frame image set X2;

[0137] S7. Form a pose data set Pose1 from the preprocessed sequence frame image set X2';

[0138] S8. Input the preprocessed sequence frame image set X2' and the pose data set Pose1 into the NSVF model, and reconstruct and generate a 3D building model through the NSVF model;

[0139] S9. Train the NSVF model by introducing a loss function to reduce the rendering loss and optimize the structure to obtain the final 3D building model;

[0140] In step S1, the structured text of the building information features includes: the building style features, building structure features, building material features, building decoration features, etc. of a certain type of building in a certain period;

[0141] Step S8 specifically includes:

[0142] S801. Divide the 3D space into sparse voxel grids through the NSVF model, and use the sparse voxel grids to represent the structure and position of the 3D building model;

[0143] S802. Input the preprocessed sequence frame image set X2' and the pose data set Pose1 into the NSVF model for feature extraction. The sparse voxel grid contains multiple sparse voxels. In each sparse voxel, use the neural network in the NSVF model to learn and extract features, and output the color and density of the voxel, thereby learning the complex details and textures on the surface of the 3D building model in the 3D space;

[0144] S803. According to the structure and position of the 3D building model obtained in step S801 and the color and density of the voxels obtained in step S802, reconstruct and generate a 3D building model through voxel sampling and volume rendering of the NSVF model;

[0145] On this basis, in this embodiment, step S8 specifically includes:

[0146] Step S9 specifically includes:

[0147] S901. Obtain the 3D building model in step S8, export the rendered images in the 3D building model to generate a set of rendered images, obtain the preprocessed sequence frame image set X2' in step S6, introduce a loss function with the goal of minimizing the difference between the set of rendered images and the preprocessed sequence frame image set X2', and train the NSVF model to reduce the rendering loss. The commonly used loss function is the mean squared error (MSE), and the formula is as follows:

[0148]

[0149] where L represents the mean squared error; N represents the total number of samples, that is, there are N data points in the dataset; i represents the number of the i-th data point in the dataset N, C i is the pixel color rendered by the model, is the pixel color of the real image;

[0150] S902. Through training the NSVF model, fill and optimize the sparse voxel grid; during the training process, the model will continuously optimize the sparsity of the sparse voxel grid and optimize the structure of the sparse voxel grid to better conform to the actual 3D structure of the object;

[0151] S903. Perform labeling processing on the materials and structures of the 3D building model and store them in layers according to the structure to facilitate the display and management of the model, and obtain the final 3D building model.

[0152] Embodiment 5

[0153] The embodiment of the present invention also provides a sparse view building 3D reconstruction device based on generative sequence frame images. This device uses the sparse view building 3D reconstruction method based on generative sequence frame images as described above, and includes:

[0154] A building information feature structured text generation module, a model training module, a building image generation module, a target building video generation module, a sequence frame image set generation module, a pose dataset generation module, and a 3D building model reconstruction module;

[0155] The building information feature structured text generation module is used to collect a certain type of building information in a certain period, and generate building information feature structured text and the corresponding building images of the text;

[0156] The model training module is used to train the LoRA model using the building information feature structured text generated by the building information feature structured text generation module and the corresponding building images of the text;

[0157] The building image generation module is used to input scene prompts to the ComfyUI platform, generate multiple building images in combination with the LoRA model trained by the model training module, and select the building images that meet the preset image standards;

[0158] The target building video generation module is used to input action prompts to the ComfyUI platform, input the building images selected by the building image generation module into the ComfyUI platform, and generate target building videos on multiple different track planes in combination with the lens track model and the AI video generation model, and select multiple target building videos on different track planes that meet the preset video standards;

[0159] The sequence frame image set generation module is used to perform video image processing on the target building videos selected by the target building video generation module to generate a sequence frame image set and preprocess the sequence frame image set;

[0160] The pose data set generation module is used to process the sequence frame image set preprocessed by the sequence frame image set generation module to generate a pose data set;

[0161] The three-dimensional building model reconstruction module is used to reconstruct a three-dimensional building model through the NSVF model using the sequence frame image set preprocessed by the sequence frame image set generation module and the pose data set generated by the pose data set generation module.

[0162] The above are only embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the scope of the application of the present invention shall be included within the protection scope of the present invention.

Claims

1. A sparse-view building three-dimensional reconstruction method based on generative sequential frame images, characterized in that It includes the following steps: S1. Collect a certain type of building information in a certain period, generate a structured text of building information features and the corresponding building images of the text, denoted as dataset X1; S2. Input dataset X1 into the Flux model base for training the LoRA model of a certain type of building in a certain period to obtain a trained building style LoRA model; S3. Input style description prompt words into the ComfyUI platform, generate multiple building images in combination with the trained building style LoRA model, and select a building image that meets the preset image standards; S4. Input subject and action description prompt words into the ComfyUI platform, and input the building image selected in step S3 into the ComfyUI platform. Combine the lens track class model and the AI video generation class model to generate target building videos with multiple different track planes, and select target building videos with multiple different track planes that meet the preset video standards; S5. Perform video image processing on the multiple target building videos with different track planes selected in step S4 to generate multiple sequence frame images corresponding to the target building videos with multiple track planes, and integrate the multiple sequence frame images to form a sequence frame image set X2; S6. Preprocess the sequence frame image set X2; S7. Form a pose dataset Pose1 from the preprocessed sequence frame image set X2'; S8. Input the preprocessed sequence frame image set X2' and the pose dataset Pose1 into the NSVF model, and reconstruct and generate a 3D building model through the NSVF model; S9. Train the NSVF model by introducing a loss function to reduce rendering loss and optimize the structure to obtain the final 3D building model.

2. The sparse-view building 3D reconstruction method based on generative sequence frame images according to claim 1, wherein In step S1, the structured text of building information features includes: building style features, building structure features, building material features, and building decoration features of a certain type of building in a certain period.

3. The sparse view building 3D reconstruction method based on generative sequence frame images according to claim 2, characterized in that, In step S5, perform video image processing on the selected target building videos with multiple different track planes by using the method of extracting frames with OpenCV.

4. The sparse view building three-dimensional reconstruction method based on generative sequence frame images according to claim 2, wherein In step S6, preprocess each frame image in the sequence frame image set X2, and the preprocessing methods include denoising, brightness equalization, and contrast enhancement.

5. The sparse view building three-dimensional reconstruction method based on generative sequence frame images according to claim 2, wherein In step S7, calculate the camera pose of the preprocessed sequence frame image set X2', and form a pose dataset Pose1 through the SfM algorithm in computer vision.

6. The sparse view building three-dimensional reconstruction method based on generative sequence frame images according to claim 5, characterized in that, The steps included in the SfM algorithm are camera projection model, feature matching and initial pose estimation, triangulation, and bundle adjustment.

7. The sparse-view building 3D reconstruction method based on generative sequence frame images according to claim 2, wherein Step S8 specifically includes: S801. Divide the 3D space into sparse voxel grids through the NSVF model, and use the sparse voxel grids to represent the structure and position of the 3D building model; S802. Input the preprocessed sequence frame image set X2' and the pose dataset Pose1 into the NSVF model for feature extraction. The sparse voxel grid contains multiple sparse voxels. In each sparse voxel, use the neural network in the NSVF model to learn and extract features, and output the color and density of the voxel, so as to learn the complex details and textures on the surface of the 3D building model in the 3D space. S803. Based on the structure and position of the three-dimensional building model obtained in step S801 and the color and density of the voxels obtained in step S802, a three-dimensional building model is reconstructed through voxel sampling and volume rendering of the NSVF model.

8. The sparse view building three-dimensional reconstruction method based on generative sequence frame images according to claim 7, characterized in that, Step S9 specifically includes: S901. Obtain the three-dimensional building model in step S8, export the rendered images in the three-dimensional building model to generate a set of rendered images, obtain the preprocessed sequence frame image set X2' in step S6, introduce a loss function with the goal of minimizing the difference between the set of rendered images and the preprocessed sequence frame image set X2', and train the NSVF model to reduce the rendering loss; S902. Through training the NSVF model, fill and optimize the sparse voxel grid; S903. Perform labeling processing on the materials and structures of the three-dimensional building model, and store them in layers according to the structure to obtain the final three-dimensional building model.

9. A sparse-view building three-dimensional reconstruction device based on generative sequential frame images, characterized in that, Using the sparse-view building three-dimensional reconstruction method based on generative sequence frame images as described in any one of claims 1-8, including: A building information feature structured text generation module, a model training module, a building image generation module, a target building video generation module, a sequence frame image set generation module, a pose data set generation module, and a three-dimensional building model reconstruction module; The building information feature structured text generation module is used to collect a certain type of building information in a certain period, and generate building information feature structured text and the corresponding building images of the text; The model training module is used to train the LoRA model with the building images corresponding to the building information feature structured text generated by the building information feature structured text generation module; The building image generation module is used to input scene prompts to the ComfyUI platform, generate multiple building images in combination with the LoRA model trained by the model training module, and select the building images that meet the preset image criteria; The target building video generation module is used to input action prompts to the ComfyUI platform, input the building images selected by the building image generation module into the ComfyUI platform, and generate target building videos on multiple different track planes in combination with a lens track class model and an AI video generation class model, and select target building videos on multiple different track planes that meet the preset video criteria; The sequence frame image set generation module is used to perform video image processing on the target building videos selected by the target building video generation module to generate a sequence frame image set, and preprocess the sequence frame image set; The pose data set generation module is used to process the preprocessed sequence frame image set by the sequence frame image set generation module to generate a pose data set; The three-dimensional building model reconstruction module is used to reconstruct a three-dimensional building model through the NSVF model with the preprocessed sequence frame image set by the sequence frame image set generation module and the pose data set generated by the pose data set generation module.

Images