Intelligent scenic spot three-dimensional image rendering method

Abstract

The invention relates to the field of 3D rendering, in particular to an intelligent scenic area three-dimensional image rendering method, which introduces a differentiable discrete decision into 3D fusion, supports end-to-end learning of a k value, performs discrete-continuous optimization based on an activation function, predicts an optimal k value of each voxel, introduces feature adaptive fusion based on dynamic neighborhood bidirectional retrieval, and realizes the 3D image rendering of the scenic area. The alignment of the color image and the point cloud is enhanced, the false detection rate of a small target is reduced, the detail reconstruction capability of a large target is improved, and the comprehensive rendering capability of a scenic spot is improved; according to the method, a lightweight grid is adopted to express a scenic spot subject, residual Gaussian is introduced to supplement high-frequency detail features, the number of Gaussian is reduced, rendering efficiency and capability are improved, textures are generated based on initial rendering back projection, fuzzy view angle dependence is avoided, hierarchical clustering and contour extraction from bottom to top are adopted on the basis, and the method is more efficient and efficient. The point cloud vertical structure change is dynamically detected, the point cloud is complemented, the accurate contour is extracted, the number of grid vertexes is reduced, and the rendering integrity is improved.

Description

Technical Field

[0001] The present invention belongs to the field of 3D rendering, and in particular to an intelligent scenic area three-dimensional image rendering method. Background Art

[0002] Scenic area 3D image rendering uses computer graphics technology to model, render, and present scenic areas to create realistic 3D scenes. Existing 3D rendering methods suffer from insufficient context for large objects and noise interference for small objects when modeling scenic areas. Furthermore, traditional mapping can cause blurring due to occlusion and viewpoint dependence. 3D Gaussian rendering also overuses Gaussians in simple areas, resulting in wasted computation. Summary of the Invention

[0003] In view of the above situation, in order to overcome the defects of the existing technology, the present invention provides an intelligent 3D image rendering method for scenic spots. In view of the problems of insufficient context of large targets and noise interference of small targets in existing 3D rendering when modeling scenic spots, the present invention introduces differentiable discrete decision-making into 3D fusion, supports end-to-end learning of k values, predicts the optimal k value of each voxel through discrete-continuous optimization based on Gumbel-Softmax, and introduces feature adaptive fusion based on dynamic neighborhood bidirectional retrieval, enhances the alignment of 2D color images and 3D point clouds, reduces the false detection rate of dense small targets, and increases the detail reconstruction ability of large targets. The present invention adopts lightweight mesh to express the main body of scenic spot, and introduces residual Gaussian to supplement high-frequency detail features, reduces the number of Gaussian, improves rendering efficiency and capability, generates texture based on 3DGS initial rendering and reverse projection, avoids perspective-dependent blur, and adopts bottom-up hierarchical clustering and Alpha-Shape contour extraction method on this basis to dynamically detect vertical structure changes of point cloud and complete point cloud, extract accurate contour, reduce the number of mesh vertices, and improve rendering integrity.

[0004] The present invention provides an intelligent scenic area three-dimensional image rendering method, which specifically includes the following steps:

[0005] Step S1: Image acquisition, by acquiring images of the scenic area from n perspectives, obtaining a 2D image, and constructing a 3D point cloud of the scenic area;

[0006] Step S2: Image preprocessing, preprocessing and feature extraction of 2D images and 3D point clouds respectively to obtain 2D features With 3D features , specifically including the following steps:

[0007] Step S21: completing the scene point cloud, completing the 3D point cloud to obtain a completed 3D point cloud, specifically including the following steps:

[0008] Step S211: Vertical stratification: 3D divides the point cloud into L layers along the vertical direction, and then performs completion processing from the bottom layer (i=1) to the top layer (i=L);

[0009] Step S212: Clustering and contour extraction: Use DBSCAN clustering on each layer of point set to obtain subsets, and apply the Alpha-Shape algorithm to extract contours for each subset: ;

[0010] Where, represents a subset, Represents the outline, It is the preset parameter to control the tightness of the contour;

[0011] Step S213: Dominant contour selection: if the area of ​​the current contour is smaller than that of the previous contour by more than a preset threshold, it is retained as the new dominant contour. ;

[0012] Step S214: Proxy geometry generation: stretching the dominant contour into a polyhedron along the vertical direction, filling the missing area and sampling to complete the 3D point cloud to obtain the completed 3D point cloud;

[0013] Step S22: After voxelizing the completed 3D point cloud, extract 3D voxel features through a sparse convolutional network to obtain 3D features ;

[0014] Step S23: generating a sky mask by unsupervised semantic segmentation of the 2D image and filtering out invalid areas, performing feature extraction on the segmented 2D image to obtain primary 2D features;

[0015] Step S24: Using the camera parameters and the depth information of the point cloud, the primary 2D features are projected into the 3D space and compared with the 3D features. Align and get 2D features ;

[0016] Step S3: Feature fusion, 2D features With 3D features Perform feature adaptive fusion based on dynamic neighborhood bidirectional retrieval to obtain fusion features, which specifically includes the following steps:

[0017] Step S31: 2D features Perform dynamic neighborhood retrieval with 3D as the target modality to obtain attention weights , specifically including the following steps:

[0018] Step S311: Attention mechanism processing, 2D features Perform attention mechanism processing to obtain the query vector;

[0019] Step S312: For each non-zero query vector, the size of the candidate neighborhood is predicted by the multi-layer perceptron MLP to obtain the candidate neighborhood : ;

[0020] Where, represents the activation function for prediction, represents multi-layer perceptron processing, represents a non-zero query vector;

[0021] Step S313: Use Straight-Through Estimator (STE) to estimate the candidate neighborhood Preserve the gradient and get the candidate neighborhood , ;

[0022] Step S314: According to Search for Top-k neighbor features from 3D modalities and construct a bond matrix: ;

[0023] Where, Represents the splicing operation, is the Top-k neighborhood feature, represents the bond matrix;

[0024] Step S315: Fill and normalize the key matrix to obtain attention weights : ;

[0025] Where, represents normalization, Represents a fill operation;

[0026] Step S32: 3D features As input, perform dynamic neighborhood retrieval with 2D as the target modality and repeat steps S311-S315 to obtain the attention weight ;

[0027] Step S33: Bidirectional fusion, based on attention weight and attention weights For 2D features With 3D features Perform fusion and obtain fusion features: ;

[0028] Where, represents fusion features;

[0029] Step S4: construct an initial Gaussian field and initialize Gaussian parameters, including Gaussian center position, covariance matrix, color, and transparency;

[0030] Step S5: Proxy mesh and texture generation: Use Poisson reconstruction to generate a proxy mesh M = {V, F, T} from the completed 3D point cloud. The mesh parameters represent the vertex, face, and UV coordinates, respectively. Use the initial Gaussian field to render the occlusion-independent proxy mesh, and generate the initial texture through backprojection. The initial texture is smoothed to obtain the proxy texture.

[0031] Step S6: Residual Gaussian construction: calculate the color difference between the proxy grid and the real image pixel by pixel to obtain a color residual map, and perform Gaussian importance scoring based on the color residual map and the initial Gaussian parameters. The Gaussian with a Gaussian importance score greater than the preset threshold is retained as the residual Gaussian. The Gaussian importance score is calculated as follows: ;

[0032] Where, Represents the Gaussian parameter set Gaussian parameter set, Represents the color residual image pixel The color difference value at Represents transparency, represents the transmittance, represents the Gaussian importance score;

[0033] Step S7: Joint rendering, use the initial Gaussian field to render the fused features to obtain a 3D Gaussian, project the 3D Gaussian to the projection views of n 2D images, and calculate the pixel color: ;

[0034] Where, represents the pixel color, Represents overlapping pixels in the projected view Gaussian set of Represents the color and transparency in the projection view;

[0035] The completed 3D point cloud is jointly rendered based on pixel color, proxy texture, and residual Gaussian to obtain a scenic area rendering. The rendering calculation of the point cloud is as follows: ;

[0036] Where, represents the transmittance in the residual Gaussian, is the pixel color calculated above, represents the transparency of the residual Gaussian, is the proxy mesh transmittance, The rendered color of the proxy mesh.

[0037] The beneficial results achieved by the present invention using the above scheme are as follows:

[0038] (1) In view of the problems of insufficient context for large objects and noise interference for small objects in existing three-dimensional rendering when modeling scenic spots, the present invention introduces differentiable discrete decision-making into 3D fusion, supports end-to-end learning of k values, predicts the optimal k value for each voxel through discrete-continuous optimization based on Gumbel-Softmax, and introduces feature adaptive fusion based on dynamic neighborhood bidirectional retrieval to enhance the alignment of 2D color images and 3D point clouds, reduce the false detection rate of dense small objects, increase the ability to reconstruct details of large objects, and improve the comprehensive rendering capability of scenic spots;

[0039] (2) In order to solve the problem that traditional mapping causes blurring due to occlusion and perspective dependence, and three-dimensional Gaussian rendering causes computational waste by excessive use of Gaussians in simple areas, the present invention adopts lightweight grids to express the main body of the scenic area, and introduces residual Gaussians to supplement high-frequency detail features, reduce the number of Gaussians, and improve rendering efficiency and capabilities. Textures are generated based on the reverse projection of the 3DGS initial rendering to avoid perspective-dependent blurring. On this basis, bottom-up hierarchical clustering and Alpha-Shape contour extraction methods are used to dynamically detect vertical structural changes in the point cloud and complete the point cloud, extract accurate contours, reduce the number of grid vertices, and improve rendering integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of an intelligent scenic area three-dimensional image rendering method provided by the present invention;

[0041] Figure 2 Schematic diagram of the process of feature adaptive fusion based on dynamic neighborhood bidirectional retrieval.

[0042] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] Example 1, see Figure 1 , an intelligent scenic spot three-dimensional image rendering method, specifically comprising the following steps:

[0045] Step S1: Collect 2D images of the scenic area from n perspectives and construct a 3D point cloud of the scenic area;

[0046] Step S2: Preprocess and extract features of 2D images and 3D point clouds respectively to obtain 2D features With 3D features ;

[0047] Step S3: 2D features With 3D features Perform feature adaptive fusion based on dynamic neighborhood bidirectional retrieval to obtain fusion features;

[0048] Step S4: construct an initial Gaussian field and initialize Gaussian parameters, including color, transparency, and transmittance;

[0049] Step S5: Proxy mesh and texture generation, using Poisson reconstruction to generate a proxy mesh from the completed 3D point cloud, and using the initial Gaussian field to render the occlusion-independent proxy mesh;

[0050] Step S6: Calculate the color difference between the proxy grid and the real image pixel by pixel to obtain a color residual map, and perform Gaussian importance scoring based on the color residual map and the initial Gaussian parameters. The Gaussian with a Gaussian importance score greater than a preset threshold is retained as the residual Gaussian.

[0051] Step S7: Joint rendering, using the initial Gaussian field to render the fused features to obtain a 3D Gaussian, and projecting the 3D Gaussian onto the projection views of n 2D images to calculate the pixel color: ;

[0052] Where, represents the pixel color, Represents overlapping pixels in the projected view Gaussian set of Represents the color and transparency in the projection view;

[0053] The completed 3D point cloud is jointly rendered based on pixel color, proxy texture, and residual Gaussian to obtain a scenic area rendering. The rendering calculation of the point cloud is as follows: ;

[0054] Where, represents the transmittance in the residual Gaussian, is the pixel color calculated above, represents the transparency of the residual Gaussian, is the proxy mesh transmittance, The rendered color of the proxy mesh.

[0055] By performing the above operations, the present invention uses lightweight grids to express the main body of the scenic area, and introduces residual Gaussians to supplement high-frequency detail features, reducing the number of Gaussians, improving rendering efficiency and capabilities, and generates textures based on the reverse projection of the 3DGS initial rendering to avoid perspective-dependent blur. On this basis, the present invention adopts bottom-up hierarchical clustering and Alpha-Shape contour extraction methods to dynamically detect vertical structural changes in the point cloud and complete the point cloud, extract accurate contours, reduce the number of grid vertices, and improve rendering integrity.

[0056] Example 2, see Figure 2 This embodiment is based on the above embodiment, and step S3 specifically includes the following steps:

[0057] Step S31: 2D features Perform dynamic neighborhood retrieval with 3D as the target modality to obtain attention weights ;

[0058] Step S32: 3D features As input, perform dynamic neighborhood retrieval with 2D as the target modality and repeat step S32 to obtain the attention weight ;

[0059] Step S33: Bidirectional fusion, based on attention weight and attention weights For 2D features With 3D features Perform fusion and obtain fusion features: ;

[0060] Where, Represents fusion features.

[0061] Embodiment 3: This embodiment is based on the above embodiment, and step S31 specifically includes the following steps:

[0062] Step S311: Annotate 2D features Perform attention mechanism processing to obtain the query vector;

[0063] Step S312: For each non-zero query vector, the size of the candidate neighborhood of the query vector is predicted by the multi-layer perceptron MLP to obtain the candidate neighborhood : ;

[0064] Where, represents the activation function for prediction, represents multi-layer perceptron processing, represents a non-zero query vector;

[0065] Step S313: For candidate neighborhoods Preserve the gradient and get the candidate neighborhood , ;

[0066] Step S314: According to Search for Top-k neighbor features from 3D modalities and construct a bond matrix: ;

[0067] Where, Represents the splicing operation, is the Top-k neighborhood feature, represents the bond matrix;

[0068] Step S315: Fill and normalize the key matrix to obtain attention weights .

[0069] By performing the above operations, the present invention introduces differentiable discrete decision-making into 3D fusion, supports end-to-end learning of k values, predicts the optimal k value for each voxel through discrete-continuous optimization based on Gumbel-Softmax through MLP, and introduces feature adaptive fusion based on dynamic neighborhood bidirectional retrieval, enhances the alignment of 2D color images and 3D point clouds, reduces the false detection rate of dense small targets, increases the ability to reconstruct details of large targets, and improves the comprehensive rendering capability of scenic spots.

[0070] Embodiment 4: This embodiment is based on the above embodiment, and step S2 specifically includes the following steps:

[0071] Step S21: completing the scene point cloud, performing point cloud completion on the 3D point cloud to obtain a completed 3D point cloud;

[0072] Step S211: Vertical stratification: 3D divides the point cloud into L layers along the vertical direction, and then performs completion processing from the bottom layer (i=1) to the top layer (i=L);

[0073] Step S212: Clustering and contour extraction: Use DBSCAN clustering on each layer of point set to obtain subsets, and apply the Alpha-Shape algorithm to extract contours for each subset: ;

[0074] Where, represents a subset, Represents the outline, It is the preset parameter to control the tightness of the contour;

[0075] Step S213: Dominant contour selection: if the area of ​​the current contour is smaller than that of the previous contour by more than a preset threshold, it is retained as the new dominant contour. ;

[0076] Step S214: Proxy geometry generation: stretching the dominant contour into a polyhedron along the vertical direction, filling the missing area and sampling to complete the 3D point cloud to obtain the completed 3D point cloud;

[0077] Step S22: After voxelizing the completed 3D point cloud, extract 3D voxel features through a sparse convolutional network to obtain 3D features ;

[0078] Step S23: generating a sky mask by unsupervised semantic segmentation of the 2D image and filtering out invalid areas, performing feature extraction on the segmented 2D image to obtain primary 2D features;

[0079] Step S24: Using the camera parameters and the depth information of the point cloud, the primary 2D features are projected into the 3D space and compared with the 3D features. Align and get 2D features .

[0080] Example 5: This example is based on the above example. The data collection in step S1 specifically includes the following steps:

[0081] Collect multi-view scenic area images, including high-definition photos (resolution ≥ 1920×1080) from at least 6 perspectives, covering the main areas of the scenic area (entrances, landmark buildings, roads), and ensuring more than 30% overlap to support multi-view matching;

[0082] 3D point cloud construction, dense point cloud generated by LiDAR scanning or motion recovery structure, point density ≥ 10 points / square meter;

[0083] Auxiliary data recording, including camera intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (rotation matrix, translation vector), as well as distortion coefficients (k1, k2, p1, p2).

[0084] Example 6: This example is based on the above example, and some details of the data preprocessing in step S2 are as follows:

[0085] Use a deep learning-based semantic segmentation model (Mask2Former) to segment the sky area of ​​the 2D image and generate a binary mask;

[0086] Statistical outlier filtering is used to remove noise points from the 3D point cloud and perform voxelization.

[0087] ResNet101-DCN (deformable convolution) is used as the image encoder to enhance the feature extraction capability of irregular scenic area structures. The weights are fine-tuned based on the ImageNet-22K pre-training to adapt to scenic area scenes.

[0088] Project 2D features into 3D space through camera parameters and use bilinear interpolation to solve the coordinate alignment problem;

[0089] Example 7: This example is based on the above example, and some details of the Gaussian initialization in step S3 are as follows:

[0090] Parameter generation

[0091] Use a 3D convolutional network to predict Gaussian parameters from fused features:

[0092] Position: Sigmoid activation is constrained within the voxel grid;

[0093] Covariance: positive definiteness guaranteed by Cholesky decomposition;

[0094] Color c: RGB value after tanh activation;

[0095] Transparency: Sigmoid activation;

[0096] Anchor point supplement

[0097] For sparse areas of the point cloud (density < 5 points / m³), non-zero voxel centers are added as auxiliary anchors.

[0098] Example 8, optimization of actual technical implementation:

[0099] Large-scale scenic area scene optimization:

[0100] Adopting a block processing strategy (dividing the scenic area into 50m×50m grids);

[0101] Use LOD technology to dynamically load details;

[0102] Dynamic object processing:

[0103] Detecting moving objects (such as tourists) through optical flow;

[0104] Enable transient reconstruction mode for dynamic areas (reduce the number of Gaussians);

[0105] Lighting consistency:

[0106] Add illumination invariance terms to the loss function.

[0107] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0108] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0109] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An intelligent scenic area three-dimensional image rendering method, characterized by: The specific steps include: Step S1: Collect 2D images of the scenic area from n perspectives and construct a 3D point cloud of the scenic area; Step S2: Preprocess and extract features of 2D images and 3D point clouds respectively to obtain 2D features With 3D features ; Step S3: 2D features With 3D features Perform feature adaptive fusion based on dynamic neighborhood bidirectional retrieval to obtain fusion features; Step S4: construct an initial Gaussian field and initialize Gaussian parameters, including color, transparency, and transmittance; Step S5: Proxy mesh and texture generation, using Poisson reconstruction to generate a proxy mesh from the completed 3D point cloud, and using the initial Gaussian field to render the occlusion-independent proxy mesh; Step S6: Calculate the color difference between the proxy grid and the real image pixel by pixel to obtain a color residual map, and perform Gaussian importance scoring based on the color residual map and the initial Gaussian parameters. The Gaussian with a Gaussian importance score greater than a preset threshold is retained as the residual Gaussian. Step S7: Joint rendering, use the initial Gaussian field to render the fused features to obtain a 3D Gaussian, project the 3D Gaussian to the perspective of n 2D images, obtain the projected view and calculate the pixel color: ； Where, represents the pixel color, Represents overlapping pixels in the projected view Gaussian set of Represents the color and transparency in the projection view; The completed 3D point cloud is jointly rendered based on pixel color, proxy texture, and residual Gaussian to obtain a scenic area rendering. The rendering calculation of the point cloud is as follows: ； Where, represents the transmittance in the residual Gaussian, is the pixel color, represents the transparency of the residual Gaussian, is the proxy mesh transmittance, is the rendered color of the proxy mesh, For the rendering result.

2. The intelligent scenic area three-dimensional image rendering method according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: 2D features Perform dynamic neighborhood retrieval with 3D as the target modality to obtain attention weights ; Step S32: 3D features As input, perform dynamic neighborhood retrieval with 2D as the target modality and repeat step S32 to obtain the attention weight ; Step S33: Bidirectional fusion, based on attention weight and attention weights For 2D features With 3D features Perform fusion and obtain fusion features: ； Where, Represents fusion features.

3. The intelligent scenic area three-dimensional image rendering method according to claim 2, characterized in that: Step S31 specifically includes the following steps: Step S311: Annotate 2D features Perform attention mechanism processing to obtain the query vector; Step S312: For each non-zero query vector, the size of the candidate neighborhood of the query vector is predicted by the multi-layer perceptron MLP to obtain the candidate neighborhood : ； Where, represents the activation function for prediction, represents multi-layer perceptron processing, represents a non-zero query vector; Step S313: For candidate neighborhoods Preserve the gradient and get the candidate neighborhood , ; Step S314: According to Search for Top-k neighbor features from 3D modalities and construct a bond matrix: ； Where, Represents the splicing operation, is the Top-k neighborhood feature, represents the bond matrix; Step S315: Fill and normalize the key matrix to obtain attention weights .

Images