New view synthesis method fusing local spatial information

Abstract

The application provides a new view synthesis method fusing local space information, and relates to the technical field of view synthesis.The application uses a feature golden tower network to obtain picture features fusing different scale information of a scene when forming a neural point cloud, adjusts the sampling position of local features through a neural point cloud feature alignment module, queries the picture feature area corresponding to each point cloud and generates a neural point cloud.Compared with directly using a 2D convolution network for down-sampling, the application can better focus on the relevant area and capture more useful features, and integrate the features into the neural point cloud.Before a volume density prediction network, the application captures the context representation of the local space through a neural point cloud transformer module, and the local neural point cloud features complete the self clues through a self-attention mechanism, so that the sparsity and irregularity of the point cloud can be effectively processed, and the quality of the final synthesized picture is improved.The synthesis result of the application in a real scene and a weak texture area is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of perspective synthesis technology, specifically to a novel perspective synthesis method that integrates local spatial information. Background Technology

[0002] Synthesizing novel scene images from a set of 2D images and their associated photographic poses is an important topic in computer vision. Mildenhall et al. (2020) proposed a neural rendering method—neuralradiance field (NeRF)—which uses a fully connected multi-layer perception (MLP) to store real scene information and represents the scene as an implicit neural network. NeRF, as a novel scene representation method, has significantly improved visual performance in rendering tasks. Although NeRF synthesizes high-quality images, this method sacrifices training time, as each sampling point needs to pass through the entire MLP model during scene reconstruction. To alleviate the slow scene reconstruction speed of the original NeRF, Yu et al. (2021) introduced a hierarchical data structure called PlenOctrees, which incorporates voxels. This structure uses octrees to avoid oversampling and reduces sampling points with low contribution, enabling real-time image rendering compared to NeRF. Reiser et al. (2021) proposed a method that does not require... Unlike large MLPs, the voxel-based method (Kilo-NeRF) uses thousands of small MLPs to represent scenes, each responsible for describing a local part of the scene. This divide-and-conquer strategy improves computation speed by about three orders of magnitude. Fridovich-Keil et al. (2022) completely abandoned neural networks, instead using sparse voxel grids to store spherical harmonic functions to represent scenes, and obtained the RGB values ​​and volume density of sampling points through trilinear interpolation, improving training speed by nearly two orders of magnitude. Müller et al. (2022) proposed a multi-resolution hashing encoding method, using CUDA programming to reduce computation time, thereby improving scene reconstruction efficiency; training a high-quality scene only takes a few seconds.

[0003] On the other hand, many scholars have conducted research on using NeRF to improve the quality of scene reconstruction. To address the issues of blurry and jagged images, Barron et al. (2021) projected a cone from the camera origin to the "pixel" to replace the light rays in NeRF, rendering the pixel using a Gaussian distribution that approximates the cone-shaped truncated cone corresponding to that pixel. Verbin et al. (2022) replaced the parameterized view-dependent outgoing radiance of NeRF with reflected radiance, constructing the function using a set of spatially varying scene properties. Compared to the original NeRF, this method can additionally output surface normal, diffuse color, specular weight, and roughness, calculate the specular color based on the viewpoint direction and roughness, and predict the final color using the diffuse color and specular weight. Wang et al. (2023) modeled surface reflection as a neural network, taking the direction of the incident light rays and the lighting conditions in the scene as inputs, and outputting the color and intensity of the surface reflection. By optimizing network parameters using rendered images with lighting and material information during training, the network can learn the reflection behavior of complex materials, including the optical properties of various materials such as metals, plastics, and skin.

[0004] Despite the high quality of images synthesized from neural radiation fields, several challenges remain when only a limited number of scene views are available (Chen et al., 2021; Yu et al., 2021; Jain et al., 2021). Numerous researchers have attempted to address this issue through various methods, including content-based regularization (Jain et al., 2021), patch-based regularization (Niemeyer et al., 2022), utilizing image features (Yu et al., 2021), based on diffusion prior information (Deng et al., 2023; Wynn and Turmukhambetov, 2023), methods based on deep supervision (Xiao et al., 2024; Liu et al., 2024), and methods based on Gaussian splatting (Chung et al., 2024). Yu et al. (2021) proposed a pixelNeRF network, enabling NeRF to be trained in different scenes. By learning prior knowledge of the scene, it can reconstruct the scene using only one or a few images. Chen et al. (2021) used a pre-trained convolutional network to extract image features and mapped these features onto the scanning plane of a reference view, thereby constructing a cost volume based on three-dimensional voxels. They then used a 3D convolutional network to reconstruct the neural coding volume and employed an MLP to decode the implicit features within the neural coding volume into volume density and color values. Xu et al. (2022) reconstructed complex visual scenes using only a single-view image to train the neural radiation field. They constructed a semi-supervised learning framework, completing scene reconstruction by designing supervision signals based on geometry and semantics. Geometric supervision ensured geometric consistency across multiple views through image deformation and camera pose parameters. Semantic supervision extracted image features using a pre-trained ViT and compared them with the overall structure to optimize semantic quality under unknown views. Xiao et al. (2024) used a depth-supervised strategy, allocating more sampling points to the object surface and introducing light loss under unknown views, resulting in a more refined final result. Liu et al. (2024) used a depth prediction network to calculate the standard deviation between the predicted depth value and the sparse depth value to supervise the entire network, thereby solving the problem of sparse viewpoint input. Chung et al. (2024) used a large number of Gaussian rendering representations of the 3D scene and used depth maps to regularize the optimization process of Gaussian rendering, thereby effectively reducing the "floating artifacts" in the results.

[0005] Another approach to addressing the sparse viewpoint problem is to utilize point cloud information from sensors or photogrammetry. Xu et al. (2022) used input sparse viewpoint images to generate point clouds, and then combined the point cloud features with sampling point information to input the neural radiation field for scene reconstruction. Kulhanek and Sattler et al. (2023) meshed the input point cloud and introduced a tetrahedral structure to assist sampling points in interpolation. Zhang et al. (2023) optimized the process by transforming the point cloud to the frequency domain using Fourier functions. Govindarajan et al. (2024) proposed the Lagrange hash method, which is a neural radiation field representation method that combines point clouds and hierarchical hash tables.

[0006] The point cloud-based neural radiation field method is a novel approach for reconstructing scenes from sparse viewpoint images. It leverages the geometric priors of the point cloud scene to improve network sampling efficiency, effectively addressing the issue of excessively long NeRF training time. However, since this method requires additional point cloud input, the quality of the final synthesized new viewpoint image is limited by the accuracy, location distribution, and density of the point cloud. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to propose a novel perspective synthesis method that integrates local spatial information, comprising:

[0008] Step 1: Based on the neural point cloud generation network, process multiple sets of images to obtain the final neural point cloud P = (p o ,γ o ,f o ), where each group of images corresponds to a viewpoint, and each group of images contains multiple target images from the same viewpoint, where γ o p represents the confidence level of the point cloud. o f represents the three-dimensional position coordinates. o Features of neural point clouds;

[0009] Step 1.1: Process multiple sets of images through a deep prediction network to obtain a dense point cloud. The dense point cloud contains multiple point clouds, and each point cloud corresponds to a cloud confidence score and three-dimensional position coordinates.

[0010] Step 1.2: Using the neural point cloud feature alignment module, process the dense point cloud and multiple sets of images to obtain the final neural point cloud P = (p o ,γ o ,f o );

[0011] Step 2: Based on the neural point cloud, construct the neural radiation field network for the final neural point cloud P = (p o ,γ o ,f oThe sampling points are processed to obtain the final volume density σ and RGB value c of each sampling point. The sampling points are obtained by sampling multiple times on the line connecting each grid in the initial grid of the view to be synthesized to the center of the camera. The camera is set in a fixed position and has a viewing direction. The initial grid of the view to be synthesized is set in a fixed position.

[0012] Step 3: Based on the final volume density σ and RGB value c of each sampling point, perform volume rendering on the initial mesh to be synthesized to obtain a synthesized new perspective scene image.

[0013] Optionally, step 1.1 specifically includes:

[0014] By using a multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain a depth map for each viewpoint. Filtering and fusion operations are then performed on all depth maps to obtain a dense point cloud.

[0015] Optionally, step 1.2 specifically includes:

[0016] Step 1.2.1: Using a multi-scale feature fusion method, feature extraction is performed on each target image using the Feature Pyramid Network (FPN) structure to obtain a feature map corresponding to each target image. A deformable convolutional network is then used to process each feature map to obtain an offset feature map. The offset feature map is then meshed to obtain a meshed feature map. Two-dimensional coordinates are set for each grid in the meshed feature map, and the horizontal and vertical coordinates of the meshed feature map are in the range of [-1, 1]. Here, (-1, -1) corresponds to the grid at the upper left corner of the meshed feature map, and (1, 1) corresponds to the grid at the lower right corner of the meshed feature map.

[0017] Step 1.2.2: Based on the camera pose parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the size of the grid in the gridded feature map. Set two-dimensional coordinates for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1]. Wherein, (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane.

[0018] Step 1.2.3: Based on the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to that grid. o and three-dimensional position coordinates p o And obtain the image features in the grid, i.e., the neural point cloud features f. o , confidence level γ of point cloud o3D position coordinates p o and neural point cloud features f o By performing feature concatenation, the final neural point cloud P = (p o ,γ o ,f o ).

[0019] Optionally, step 2 specifically includes:

[0020] Step 2.1: Using the Neural Point Cloud Transformer module, determine the neural point cloud corresponding to the sampling point. Process the neural point cloud corresponding to the sampling point and the 3D coordinates of each sampling point to obtain a new feature vector {f} of the neural point cloud. i ′ ,x |i=1,...,K}, where f i ′ ,x Let K be the new feature of the i-th neural point cloud at sampling point x, where K is the number of neural point clouds corresponding to sampling point x.

[0021] Step 2.2: Develop a new feature vector {f} for the neural point cloud using a volume density prediction network. i ′ ,x The data is processed from |i=1,...,K} to obtain the final volume density σ of each sampling point;

[0022] Step 2.3: Calculate the RGB value c for each sampling point using the RGB prediction network.

[0023] Optionally, step 2.1 specifically includes:

[0024] For each sampling point, the K nearest neighbor algorithm is used to select the K nearest neural point clouds {P} based on the 3D coordinates of the sampling point. i |i=1,...,K}; then, for each of the K neural point clouds, the 3D coordinates of the neural point cloud are subtracted from the 3D coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud. The relative position coordinates of each neural point cloud are then mapped to a high-dimensional feature space to obtain the mapped relative position coordinates of each neural point cloud. This is specifically achieved through the following formula:

[0025] PE(p i -x)=(sin(2 0 π(p i -x)),cos(2 0 π(p i -x)),...,sin(2 L-1 π(p i -x))

[0026] ,cos(2 L-1 π(p i-x)));

[0027] Where x is the three-dimensional coordinate of the sampling point, p i Let PE(p) be the 3D coordinates of the i-th neural point cloud. i -x) represents the relative position coordinates of the i-th neural point cloud after mapping, and L is the encoding dimension control variable;

[0028] The mapped relative position coordinates PE(p) i -x) and the features f of the i-th neural point cloud i By cascading, the first intermediate feature f of the neural point cloud is obtained. e Specifically, this is achieved through the following formula:

[0029] f e =Concat(f i ,PE(p i -x));

[0030] Where Concat(·) is the feature concatenation operation.

[0031] The first intermediate feature f of the neural point cloud e The input is fed into the positional encoding layer to obtain the second intermediate feature PE(f) e Specifically, this is achieved through the following formula:

[0032] PE(f e )=(sin(2 0 πf e ),cos(2 0 πf e ),...,sin(2 L-1 πf e )

[0033] ,cos(2 L-1 πf e ));

[0034] The second intermediate feature PE(f) is obtained through MLP. e ) and the first intermediate feature f e The process is performed to obtain the encoded features f of the i-th neural point cloud. i,x Specifically, this is achieved through the following formula:

[0035] f i,x =MLP(Concat(f e ,PE(f e )));

[0036] All encoded features of neural point clouds constitute the neural point cloud feature sequence X = {f} i,x |i=1,…,K};

[0037] The neural point cloud feature sequence X is input into the attention layer to obtain a new feature vector {f′} of the neural point cloud. i,x |i=1,...,K}, specifically implemented through the following formula:

[0038] (Q,K,V)=X×(W q W k W v );

[0039]

[0040] MultiHead(Q,K,V)=Concat(head1,...,head n W O ;

[0041]

[0042] Where Q is the Query matrix, W q Here, W represents the weight parameters of the Query matrix, K is the Key matrix, and W represents the weight parameters of the Query matrix. k V represents the weight parameters of the Key matrix, V is the Value matrix, and W is the Weight parameter of the Key matrix. v d represents the weight parameters of the Value matrix. k W represents dimension. O This represents the output weight matrix, where n represents the total number of attention heads. i This represents the i-th attention head. W represents the weight parameters corresponding to the i-th attention head. q Include W represents the weight parameters corresponding to the i-th attention head. k Include W represents the weight parameters corresponding to the i-th attention head. v Include MultiHead(Q,K,V) represents the new feature vector {f} of the neural point cloud. i, ′ x |i=1,...,K}.

[0043] Optionally, step 2.2 specifically includes:

[0044] The new feature vector {f′ of the neural point cloud i,x The inputs |i=1,...,K} are fed into an MLP network to obtain the volume density {σ} of each neural point cloud. i,x |i=1,...,K}, specifically implemented through the following formula:

[0045] σ i,x =MLP(f′) i,x );

[0046] The final volume density σ of sampling point x is obtained by inverse distance weighting of the confidence score of each neural point cloud, specifically through the following formula:

[0047]

[0048] This allows us to obtain the final volume density σ at each sampling point.

[0049] Optionally, step 2.3 specifically includes:

[0050] Features f encoded from neural point clouds i,x Inverse distance weighting is performed to calculate the weight of each feature vector. The features of the K neural point clouds corresponding to the sampling points are aggregated to obtain the aggregated feature f. x Specifically, this is achieved through the following formula:

[0051]

[0052] The encoded feature PE(d) of the camera view direction d is calculated using the following formula:

[0053] PE(d)=(sin(2 0 πd),cos(2 0 πd),...,sin(2 L-1 πd)

[0054] ,cos(2 L-1 πd));

[0055] The encoded features PE(d) of the camera view direction d are combined with the aggregated features f x The concatenation is then input into the MLP network to obtain the RGB values ​​c of the sampled points, specifically achieved through the following formula:

[0056] c = MLP(Concat(f) x ,PE(d)));

[0057] This allows us to obtain the RGB value c for each sampling point.

[0058] Optionally, step 3 specifically includes:

[0059] Using a stratified sampling method to divide the interval [t] n ,t f Divide the data into N identical intervals, where t n and t f These are pre-set points on the line connecting the camera to each grid, with a sampling distance randomly selected within each interval, specifically represented by the following formula:

[0060]

[0061] Among them, t j This represents the sampling distance selected in the j-th interval, and the sampling point position x in the j-th interval is obtained. j Based on the sampling point position x of the j-th interval j Calculate the predicted color for each grid cell in the initial grid. Specifically, this is achieved through the following formula:

[0062]

[0063] in, Indicates sampling point x j RGB value at position sampling point x j Bulk density at location sampling point x m Bulk density at location, t j-1 t represents the sampling distance selected in the (j-1)th interval. m This represents the sampling distance selected in the m-th interval;

[0064] For each grid cell in the initial grid, predict the color to obtain a synthesized scene image with a new perspective;

[0065] Optional, also includes:

[0066] Step 4: Calculate the rendering loss value L between the synthesized new perspective scene image and the real image. render Specifically, this is achieved through the following formula:

[0067]

[0068] Among them, C gt The pixels of the actual image;

[0069] Introduce a sparsity loss function and calculate the sparsity loss value L. sqarse Specifically, it is expressed by the following formula:

[0070]

[0071] Where γ represents the confidence level;

[0072] Based on the rendering loss value L render and sparsity loss value L sqarse The joint loss value L is obtained. opt Specifically, it is expressed by the following formula:

[0073] L opt =L render +aL sparse ;

[0074] Where a is a fixed value, a = 2e -3 ;

[0075] Based on the joint loss value L opt The neural point cloud generation network and the neural radiation field network were trained multiple times to obtain the trained neural point cloud generation network and the trained neural radiation field network.

[0076] The beneficial effects of adopting the above technical solution are as follows:

[0077] This invention proposes a novel perspective synthesis method that integrates local spatial information. When forming neural point clouds, a feature pyramid network is used to acquire image features that integrate information from different scales of the scene. A neural point cloud feature alignment module adjusts the sampling positions of local features, queries the image feature regions corresponding to each point cloud, and generates neural point clouds. Compared to directly using 2D convolutional networks for downsampling, this method can better focus on relevant regions and capture more useful features, integrating them into the neural point cloud. Before the volume density prediction network, a neural point cloud Transformer module captures the contextual representation of local space. Local neural point cloud features complete their own cues through a self-attention mechanism, effectively handling the sparsity and irregularity of the point cloud and improving the quality of the final synthesized image. This invention significantly improves the synthesis results in realistic scenes and weakly textured regions. Although the model of this invention can complete the task of scene reconstruction using point clouds, limitations still exist. Because this invention uses the same volume rendering method as NeRF, the rendering speed is relatively slow; future work will focus on improving the model's real-time performance. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the novel perspective synthesis method that integrates local spatial information in an embodiment of the present invention.

[0079] Figure 2 This is a schematic diagram of the neural point cloud feature alignment module in an embodiment of the present invention;

[0080] Figure 3 This is a flowchart illustrating the deformable convolution algorithm in an embodiment of the present invention.

[0081] Figure 4 This is a schematic diagram of the neural radiation field network of the neural point cloud in an embodiment of the present invention;

[0082] Figure 5The above are visualization results of a comparative experiment conducted on the Synthetic Blender dataset in this embodiment of the invention. (a1) is the Ficus scene image; (a2), (a3), (a4), and (a5) are the new perspective scene images obtained from (a1) based on the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and this invention, respectively; (a6) is the real perspective scene image from (a1); (b1) is the Chair scene image; (b2), (b3), (b4), and (b5) are the new perspective scene images obtained from (b1) based on the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and this invention, respectively; (b6) is the real perspective scene image from (a1); (b7) is the Chair scene image; (b8) is the Chair scene image; (b9) is the Chair scene image; (b1) is the Chair scene image; (b2), (b3), (b4), and (b5) are the new perspective scene images obtained from (b1) based on the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and this invention, respectively ... (b1) is the real-view scene image; (c1) is the Lego scene image; (c2), (c3), (c4), and (c5) are the new-view scene images obtained by (c1) based on the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the present invention, respectively; (c6) is the real-view scene image of (c1); (d1) is the Drums scene image; (d2), (d3), (d4), and (d5) are the new-view scene images obtained by (d1) based on the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the present invention, respectively; (d6) is the real-view scene image of (d1).

[0083] Figure 6 The above are visualization comparison results of real scene images in the embodiments of the present invention. Among them, (a1) is the Barn scene image, (a2), (a3), and (a4) are the new perspective scene images obtained by (a1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (a5) is the real perspective scene image of (a1); (b1) is the Truck scene image, (b2), (b3), and (b4) are the new perspective scene images obtained by (b1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (b5) is the real perspective scene image of (b1); (c1) is the Caterpilar scene image, (c2), (c3), and (c4) are the new perspective scene images obtained by (c1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (c5) is the real perspective scene image of (c1).

[0084] Figure 7 The following are training performance diagrams for different scenarios in the embodiments of the present invention: (a) is the training performance diagram for the Lego scenario; (b) is the training performance diagram for the Ficus scenario; and (c) is the training performance diagram for the Drums scenario.

[0085] Figure 8The images shown are visualizations of the ablation experiments on the ScanNet dataset in this embodiment of the invention. (a1) is a scene image; (a2), (a3), and (a4) are new perspective scene images obtained by (a1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (a5) is the real perspective scene image of (a1); (b1) is a scene image; (b2), (b3), and (b4) are new perspective scene images obtained by (b1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (b5) is the real perspective scene image of (b1); (c1) is a scene image; (c2), (c3), and (c4) are new perspective scene images obtained by (c1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (c5) is the real perspective scene image of (c1). Detailed Implementation

[0086] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0087] To address the problems of existing technologies, the key to solving this problem lies in how to extract reliable local spatial information from limited point cloud features. This invention improves the correlation between point clouds and local image features and utilizes the attention mechanism of Transformer (Vaswani et al., 2017) to capture local neural point cloud contextual information, providing a novel perspective synthesis method that integrates local spatial information. Specifically: 1) A novel point cloud-based neural rendering method is proposed. By integrating the semantic information of the image into the point cloud features, correlation modeling and information extraction are performed on the local neural point cloud features, enabling better synthesis of new perspective scenes from sparse image input. 2) A neural point cloud feature alignment module is proposed. By adaptively adjusting the convolution kernel size, the point cloud and image features are better aligned, improving the ability of neural point clouds to express local spatial information, thereby enhancing the synthesis quality of the entire neural radiation field. 3) Before estimating the volume density, the local spatial information of the neural point cloud around the sampling point is fully utilized, providing a Transformer module structure based on neural point clouds, which can effectively improve the local details of the scene rendering image.

[0088] To address the issues of insufficient point cloud quality and feature extraction, this invention first presents a neural point cloud feature alignment module. This module aligns the features of the point cloud with those of the matching region in the image, and then fuses them to form a neural point cloud, thereby improving its local expressive power. Secondly, a neural point cloud Transformer module is proposed to fuse the contextual information of local neural point clouds. This module can still extract reliable local spatial information even when the point cloud quality is poor, effectively enhancing the synthesis quality of the point cloud neural rendering method.

[0089] This invention provides a novel perspective synthesis method that integrates local spatial information, combined with Figure 1 The overall network structure mainly consists of two parts: a neural point cloud generation network and a neural radiation field network. The neural point cloud generation network processes the input image, generating neural point clouds by aligning point clouds and extracting image features. In the neural radiation field network, the RGB values ​​and volume density of sampling points are predicted by aggregating neural point cloud features near the sampling points and camera viewpoint information. Finally, a new viewpoint scene image is synthesized using a volume rendering formula, which may include the following steps:

[0090] Step 1: Based on the neural point cloud generation network, process multiple sets of images to obtain the final neural point cloud P = (p o ,γ o ,f o ), where each group of images corresponds to a viewpoint, and each group of images contains multiple target images from the same viewpoint, where γ o p represents the confidence level of the point cloud. o f represents the three-dimensional position coordinates. o Features of neural point clouds;

[0091] Step 1.1: Process multiple sets of images through a deep prediction network to obtain a dense point cloud. The dense point cloud contains multiple point clouds, and each point cloud corresponds to a cloud confidence score and three-dimensional position coordinates.

[0092] By using a multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain a depth map for each viewpoint. Filtering and fusion operations are then performed on all depth maps to obtain a dense point cloud.

[0093] In the specific implementation process, feature extraction is first performed on the input multi-view images. After feature matching between images from different viewpoints, a cost volume is constructed using the variance values ​​of feature points from different viewpoints. A 3D convolutional network is then used to regularize the cost volume to generate a probability volume for predicting the depth map. Finally, the depth maps of each image are filtered and fused to obtain a dense point cloud. Since the probability volume can represent the probability of the point cloud on the reconstructed object surface, trilinear interpolation can be used to obtain the confidence level of the point cloud.

[0094] Step 1.2: Using the neural point cloud feature alignment module, process the dense point cloud and multiple sets of images to obtain the final neural point cloud P = (p o ,γ o ,f o );

[0095] Combination Figure 2 A schematic diagram of the neural point cloud feature alignment module is shown, which specifically includes the following steps:

[0096] Step 1.2.1: Employ a multi-scale feature fusion method, using the feature pyramid network (FPN) structure to extract features from each target image, obtaining the feature map corresponding to each target image. This processing method helps to handle the detailed information of images at different resolutions, thereby improving the generalization ability of the network model (Lin et al., 2017).

[0097] However, when using a fixed convolutional kernel to extract image features, it is difficult to effectively process features from regions with different shapes and textures because the receptive field is the same for different locations in the image. To better align point clouds and image features and improve the relevance of aligned features, the output features of the FPN network are further processed, which can be implemented based on convolution.

[0098] For a traditional standard convolution, given a 3×3 convolution kernel R∈{(-1,-1),(-1,0),...,(1,1)}, let x(p) and y(p) represent the features of the input feature map x and the output feature map y at position p, respectively, and ω n The convolution kernel weights have the following feature mappings:

[0099]

[0100] Compared to standard convolution, deformable convolution (Dai et al., 2017; Zhu et al., 2019) can learn additional offsets, causing the sampling location to spread outwards, specifically:

[0101] Deformable convolutional networks are represented by the following formula:

[0102]

[0103] In the formula, Δp n It is a learnable offset of position, a real number with an unconstrained range. For example... Figure 3 As shown, standard convolution incorporates unwanted image information into the extracted features, while deformable convolution uses a separate convolutional layer to learn the offset when extracting image features. It can adaptively adjust the size of the receptive field according to different positions and shapes of objects, and focuses on sampling the region of interest when aligning point clouds with image features.

[0104] Based on the principle of deformable convolution with standard kernels, this invention selects deformable convolution as the feature offset network. The feature offset network alters the sampling position of the convolution kernel, dynamically adjusting the receptive field of the aligned features. This allows the aligned features of the point cloud to focus on highly relevant image regions, better adapting to different scene structures and textures, and enhancing the ability of the neural point cloud to express surrounding spatial information. By learning additional offsets at the sampling positions, the feature mapping relationship of the convolution kernel is changed.

[0105] Specifically, a deformable convolutional network is used to process each feature map to obtain an offset feature map. The offset feature map is then meshed to obtain a meshed feature map. Two-dimensional coordinates are set for each grid in the meshed feature map, and the horizontal and vertical coordinates of the meshed feature map are in the range of [-1, 1]. Here, (-1, -1) corresponds to the grid at the top left corner of the meshed feature map, and (1, 1) corresponds to the grid at the bottom right corner of the meshed feature map.

[0106] Step 1.2.2: Based on the camera pose parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the size of the grid in the gridded feature map. Set two-dimensional coordinates for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1]. Wherein, (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane.

[0107] Step 1.2.3: Based on the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to that grid. o and three-dimensional position coordinates p o And obtain the image features in the grid, i.e., the neural point cloud features f. o , confidence level γ of point cloud o 3D position coordinates p o and neural point cloud features f o By performing feature concatenation, the final neural point cloud P = (p o ,γ o ,f o ).

[0108] Step 2: Combining Figure 4 Based on the neural point cloud, a neural radiation field network is used to calculate the final neural point cloud P = (p o ,γ o ,f oThe sampling points are processed to obtain the final volume density σ and RGB value c of each sampling point. The sampling points are obtained by sampling multiple times on the line connecting each grid in the initial grid of the view to be synthesized to the center of the camera. The camera is set in a fixed position and has a viewing direction. The initial grid of the view to be synthesized is set in a fixed position.

[0109] When predicting the volume density of sampling points, the features f of the neural point cloud are directly applied. i The method of inputting the volume density σ into an MLP network is not accurate enough, which can easily lead to a decrease in the quality of the output. Therefore, processing neural point cloud features alone is insufficient to predict the volume density of complex scenes. To solve this problem, a neural point cloud Transformer module is used before predicting the volume density to better capture the interrelationships and complementary information between local neural point clouds. This allows the neural point cloud features to contain more detailed cues, generating a more realistic volume density distribution and exhibiting higher detail fidelity when synthesizing images from new perspectives. This is specifically achieved through step 2.1.

[0110] Step 2.1: Using the Neural Point Cloud Transformer module, determine the neural point cloud corresponding to the sampling point. Process the neural point cloud corresponding to the sampling point and the 3D coordinates of each sampling point to obtain a new feature vector {f} of the neural point cloud. i ′ ,x |i=1,...,K}, where f i ′ ,x Let K be the new feature of the i-th neural point cloud at sampling point x, where K is the number of neural point clouds corresponding to sampling point x.

[0111] For each sampling point, the K nearest neighbor algorithm is used to select the K nearest neural point clouds {P} based on the 3D coordinates of the sampling point. i |i=1,...,K}; then, for each of the K neural point clouds, the 3D coordinates of the neural point cloud are subtracted from the 3D coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud. The relative position coordinates of each neural point cloud are then mapped to a high-dimensional feature space to obtain the mapped relative position coordinates of each neural point cloud. This is specifically achieved through the following formula:

[0112] PE(p i -x)=(sin(2 0 π(p i -x)),cos(2 0 π(p i -x)),...,sin(2 L-1 π(p i -x))

[0113] ,cos(2 L-1 π(pi -x)));

[0114] Where x is the three-dimensional coordinate of the sampling point, p i Let PE(p) be the 3D coordinates of the i-th neural point cloud. i -x) represents the relative position coordinates of the i-th neural point cloud after mapping, and L is the encoding dimension control variable;

[0115] The mapped relative position coordinates PE(p) i -x) and the features f of the i-th neural point cloud i By cascading, the first intermediate feature f of the neural point cloud is obtained. e Specifically, this is achieved through the following formula:

[0116] f e =Concat(f i ,PE(p i -x));

[0117] Where Concat(·) is the feature concatenation operation.

[0118] The first intermediate feature f of the neural point cloud e The input is fed into the position encoding layer, where it is further encoded and enlarged to capture high-frequency feature information of the neural point cloud, thus obtaining the second intermediate feature PE(f). e Specifically, this is achieved through the following formula:

[0119] PE(f e )=(sin(2 0 πf e ),cos(2 0 πf e ),...,sin(2 L-1 πf e )

[0120] ,cos(2 L-1 πf e ));

[0121] The second intermediate feature PE(f) is obtained through MLP. e ) and the first intermediate feature f e The process is performed to obtain the encoded features f of the i-th neural point cloud. i,x This approach can better fit the spatial information of the scene surrounding the sampling points, improve the network's translation invariance to the point cloud, enhance the network's ability to extract local scene features, and strengthen the generalization ability of the entire network model. Specifically, this is achieved through the following formula:

[0122] f i,x =MLP(Concat(f e ,PE(fe )));

[0123] All encoded features of neural point clouds constitute the neural point cloud feature sequence X = {f} i,x |i=1,…,K};

[0124] The neural point cloud feature sequence X is input into the attention layer to obtain a new feature vector {f} of the neural point cloud. i, ′ x |i=1,...,K}, specifically implemented through the following formula:

[0125] (Q,K,V)=X×(W q W k W v );

[0126]

[0127] MultiHead(Q,K,V)=Concat(head1,...,head n W O ;

[0128]

[0129] Where Q is the Query matrix, W q Here, W represents the weight parameters of the Query matrix, K is the Key matrix, and W represents the weight parameters of the Query matrix. k V represents the weight parameters of the Key matrix, V is the Value matrix, and W is the Weight parameter of the Key matrix. v d represents the weight parameters of the Value matrix. k W represents dimension. O This represents the output weight matrix, where n represents the total number of attention heads. i This represents the i-th attention head. W represents the weight parameters corresponding to the i-th attention head. q Include W represents the weight parameters corresponding to the i-th attention head. k Include W represents the weight parameters corresponding to the i-th attention head. v Include MultiHead(Q,K,V) represents the new feature vector {f} of the neural point cloud. i, ′ x |i=1,...,K}.

[0130] Step 2.2: Develop a new feature vector {f} for the neural point cloud using a volume density prediction network. i ′ ,xThe data is processed from |i=1,...,K} to obtain the final volume density σ of each sampling point;

[0131] The new feature vector {f} of the neural point cloud i ′ ,x The inputs |i=1,...,K} are fed into an MLP network to obtain the volume density {σ} of each neural point cloud. i,x |i=1,...,K}, specifically implemented through the following formula:

[0132] σ i,x =MLP(f′) i,x );

[0133] The final volume density σ of sampling point x is obtained by inverse distance weighting of the confidence score of each neural point cloud, specifically through the following formula:

[0134]

[0135] This allows us to obtain the final volume density σ at each sampling point.

[0136] Step 2.3: Calculate the RGB value c for each sampling point using the RGB prediction network.

[0137] Features f encoded from neural point clouds i,x Inverse distance weighting is performed to calculate the weight of each feature vector. The features of the K neural point clouds corresponding to the sampling points are aggregated to obtain the aggregated feature f. x Specifically, this is achieved through the following formula:

[0138]

[0139] The encoded feature PE(d) of the camera view direction d is calculated using the following formula:

[0140] PE(d)=(sin(2 0 πd),cos(2 0 πd),...,sin(2 L-1 πd)

[0141] ,cos(2 L-1 πd));

[0142] Since the RGB values ​​of the sampling points are related to the observation direction, the encoded feature PE(d) of the camera view direction d is combined with the aggregated feature f. x The concatenation is then input into the MLP network to obtain the RGB values ​​c of the sampled points, specifically achieved through the following formula:

[0143] c = MLP(Concat(f) x ,PE(d)));

[0144] This allows us to obtain the RGB value c for each sampling point.

[0145] This method uses an MLP network to predict the RGB values ​​of sampling points. While maintaining sufficient accuracy in the prediction results, it only requires decoding the aggregated features, which can reduce the computational cost of the network model.

[0146] The extraction process from steps 2.1 to 2.3 above can be summarized by the following formula:

[0147] (c,σ)=G(x,d,P1,...P K );

[0148] Where G represents the neural radiation field network.

[0149] Step 3: Based on the final volume density σ and RGB value c of each sampling point, perform volume rendering on the initial mesh to be synthesized to obtain a synthesized new perspective scene image.

[0150] A new perspective image of the scene was synthesized using the volume rendering method in NeRF (Kajiya and Von Herzen, 1984). Specifically, a layered sampling method was used to sample the interval [t]. n ,t f Divide the data into N identical intervals, where t n and t f These are pre-set points on the line connecting the camera to each grid, with a sampling distance randomly selected within each interval, specifically represented by the following formula:

[0151]

[0152] Among them, t j This represents the sampling distance selected in the j-th interval, and the sampling point position x in the j-th interval is obtained. j Based on the sampling point position x of the j-th interval j Calculate the predicted color for each grid cell in the initial grid. Specifically, this is achieved through the following formula:

[0153]

[0154] in, Indicates sampling point x j RGB value at position sampling point x j Bulk density at location sampling point x m Bulk density at location, t j-1t represents the sampling distance selected in the (j-1)th interval. m This represents the sampling distance selected in the m-th interval;

[0155] For each grid cell in the initial grid, predict the color to obtain a synthesized scene image with a new perspective;

[0156] Step 4: To measure the error between the output image and the ground truth image, the L2 function is used as the loss function to calculate the rendering loss value L between the synthesized new perspective scene image and the real image. render Specifically, this is achieved through the following formula:

[0157]

[0158] Among them, C gt The pixels of the actual image;

[0159] The confidence level γ represents the probability that a point cloud lies on an object's surface. To make the model more biased towards the neural point cloud features of the object's surface during training, a sparsity loss function is introduced, and the sparsity loss value L is calculated. sqarse Specifically, it is expressed by the following formula:

[0160]

[0161] Where γ represents the confidence level;

[0162] This loss function can make the confidence γ of the point cloud approach 0 or 1, thus increasing the constraint on point clouds with low contribution. During scene-by-scene optimization, the joint loss function of rendering loss and sparsity loss is used as the overall loss of the network, and a fixed value a = 2e is used. -3 To balance the impact of the sparsity loss function on the overall loss, the rendering loss value L is used as a basis. render and sparsity loss value L sqarse The joint loss value L is obtained. opt Specifically, it is expressed by the following formula:

[0163] L opt =L render +aL sparse ;

[0164] Where a is a fixed value, a = 2e -3 ;

[0165] Based on the joint loss value L opt The neural point cloud generation network and the neural radiation field network were trained multiple times to obtain the trained neural point cloud generation network and the trained neural radiation field network.

[0166] Based on the above scheme, the following experiment was conducted:

[0167] 1. Experimental Environment

[0168] The system used was Ubuntu 18.04, with an Intel Core i9-10900 CPU, 32GB of RAM, an RTX 3090 graphics card, and Python 3.7 as the programming language. The model was built and tested using the VS Code compiler.

[0169] 2. Dataset and Evaluation Metrics

[0170] Peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS) are used to evaluate the quality of image synthesis (Mildenhall et al., 2020). PSNR focuses on the pixel value error between the synthesized and real images; a higher value indicates a smaller error. SSIM evaluates the structural similarity between two images, ranging from -1 to 1; a value of 1 indicates that the two images are identical. LPIPS assesses the similarity perceived by humans, comparing visual differences between two images beyond pixel-level differences; a lower value indicates better visual quality.

[0171] Experiments were conducted using four commonly used datasets: DTU, NeRF Synthetic, Tanks and Temples, and ScanNet. DTU is an indoor scene dataset, with each scene containing 49 object shooting perspectives, each with 7 brightness levels, and an image resolution of 512×640. NeRF Synthetic is a synthetic dataset containing eight scenes, each with 100 training images and 200 test images. These images were entirely rendered and synthesized using Blender, with an image resolution of 800×800. ScanNet is an indoor scanning dataset, and experiments were conducted on two scenes, Scene-101 and Scene-241. Consistent with PointNeRF, this invention sampled 20% of the images, i.e., 1463 images from Scene-241 and 1000 images from Scene-101 for training, with the remaining images used for evaluation. Tanks and Temples is a large outdoor scene dataset containing 14 scenes; this experiment selected 5 of these scenes for testing.

[0172] 3. Experimental Details

[0173] First, the model is pre-trained on the DTU dataset, and the training and test sets are split using pixelNeRF and MVSNeRF methods. During this stage, the predicted values ​​of the rendering formula are supervised based on the actual pixel colors of real images, and the entire network is trained end-to-end using only the rendering loss. After pre-training, fine-tuning is performed on various scenes of the Synthetic Blender, Tanks and Temples, and ScanNet datasets. To better constrain the point clouds, a joint loss function L0 is adopted. opt Training is then performed. The network training employs the adaptive learning rate optimization Adam algorithm, which dynamically adjusts the learning rate to more effectively balance convergence speed and stability during training. The initial learning rate is set to 5e. -4 The decay rate parameters were set to 0.9 and 0.99, the number of iterations on the Synthetic Blender, Tanks and Temples datasets was 200k, and the number of training iterations on the ScanNet dataset was set to 300k.

[0174] 4. Experimental Results

[0175] This invention was compared with more advanced methods such as Point-NeRF, Tetra-NeRF, and FreqPCR on three datasets: NeRFSynthetic, Tanks and Temples, and ScanNet. Table 1 shows the PSNR results of different methods on the NeRF Synthetic dataset for synthetic scenes. The scene images in this dataset are all synthesized by Blender. Since the lighting environment and sharpness of the input images are generated under ideal conditions, various point cloud-based models all perform well on this dataset. This invention focuses more on improving the rendering quality degradation caused by the difficulty of obtaining point clouds in complex environments. The average PSNR for each scene reached 33.45dB, achieving the best result among several methods.

[0176] Table 1. PSNR results of different methods on the NeRF Synthetic dataset.

[0177]

[0178] In this text, bold text represents the optimal value for each line, underlined text represents the suboptimal value, and "*" indicates a point cloud-based method.

[0179] Figure 5Visualization results of comparative experiments on the Synthetic Blender dataset are presented. (a1) is the Ficus scene image; (a2), (a3), (a4), and (a5) are scene images obtained from (a1) using the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the new perspective obtained by this invention, respectively; (a6) is the real-view scene image from (a1); (b1) is the Chair scene image; (b2), (b3), (b4), and (b5) are scene images obtained from (b1) using the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the new perspective obtained by this invention, respectively; (b6) (b1) is a real-view scene image; (c1) is a Lego scene image; (c2), (c3), (c4), and (c5) are scene images of (c1) obtained using the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the new perspective obtained by this invention, respectively; (c6) is a real-view scene image of (c1); (d1) is a Drums scene image; (d2), (d3), (d4), and (d5) are scene images of (d1) obtained using the NPBG algorithm, Point-NeRF algorithm, Tetra-NeRF algorithm, and the new perspective obtained by this invention, respectively; (d6) is a real-view scene image of (d1). Figure 5 It is evident that, compared to other similar methods, this invention can better handle the problems of holes and noise caused by sparse point clouds. Furthermore, since the features of neural point clouds originate from images from different viewpoints, combining point clouds with visual images can help reduce the impact of specular reflection. After using the neural point cloud feature alignment module, the different viewpoint illumination information contained in each neural point cloud will be more accurate. In the Transformer module of the neural point cloud, correlation modeling is performed on the neural point clouds around the sampling points to estimate the contribution rate of each viewpoint feature to specular reflection. Therefore, the rendering results in specular reflection areas are improved, and the transitions in areas with similar image textures and object edges are more natural.

[0180] Tables 2 and 3 present the PSNR, SSIM, and LPIPS results of different methods on the Tanks and Temples and ScanNet datasets for real-world scenes. Due to the influence of factors such as the performance of the shooting device and ambient lighting on real-world images, the quality of synthesized point clouds is often poor. Tetra-NeRF obtains a neural radiation field represented by a tetrahedron by triangulating the point cloud and uses interpolation to obtain the features of the sampling points, making the image smoother and reducing the generation of holes, but at the same time, it loses local texture details. The neural point cloud Transformer module of this invention does not require additional operations on the point cloud. Based on the locality, location, and high-dimensional feature information of the local neural point cloud, it significantly improves the local details of the synthesized image. As shown in Table 2, this invention achieves the best results in PSNR and SSIM on the Tanks and Temples dataset, with LPIPS only lower than Tetra-NeRF. Therefore, compared with other advanced methods of the same type, this invention can more effectively process real-world images, has good generalization ability, and the synthesized image has low distortion and higher similarity to the ground truth image. Figure 6 The results are a visual comparison of the real scene images. (a1) is the Barn scene image, (a2), (a3), and (a4) are the new perspective scene images obtained by (a1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (a5) is the real perspective scene image of (a1); (b1) is the Truck scene image, (b2), (b3), and (b4) are the new perspective scene images obtained by (b1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (b5) is the real perspective scene image of (b1); (c1) is the Caterpilar scene image, (c2), (c3), and (c4) are the new perspective scene images obtained by (c1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (c5) is the real perspective scene image of (c1).

[0181] Table 2 Results of different methods on the Tanks and Temples dataset.

[0182]

[0183]

[0184] In this text, bold text represents the optimal value for each line, underlined text represents the suboptimal value, and "*" indicates a point cloud-based method.

[0185] Table 3 shows the results of different methods in two scenarios on the ScanNet dataset.

[0186]

[0187] In this text, bold text represents the optimal value for each line, underlined text represents the suboptimal value, and "*" indicates a point cloud-based method.

[0188] ScanNet is an indoor scanning dataset, a classic application scenario for point cloud-based neural radiation fields. Table 3 shows a comparative experiment between this invention and NeRF, NSVF, and Point-NeRF on this dataset. Due to the lack of texture and relative smoothness in most areas of this scene, traditional neural radiation field NeRF and the voxel-based method NSVF perform poorly on this dataset, while point cloud-based methods achieve more ideal results in this scenario. This invention outperforms Point-NeRF in average metrics (PSNR, SSIM, LPIP) on Scene-101 and Scene-204, achieving the highest scores of 30.95, 0.926, and 0.218, respectively.

[0189] Combination Figure 7 Among them, (a) is the training performance graph on the Lego scene; (b) is the training performance graph on the Ficus scene; and (c) is the training performance graph on the Drums scene. Figure 7 The visualization results show that the PSNR values ​​of this invention and Point-NeRF in the Lego, Ficus, and Drums scenes of the SyntheticBlender dataset are comparable. The results indicate that, compared to Point-NeRF, this invention performs better in initialization for different scenes during scene fine-tuning. This invention also achieves good experimental results with fewer iterations. After the same number of iterations, the PSNR value of the synthesized image from this invention is higher, demonstrating that this invention can more effectively fit different scenes during scene fine-tuning.

[0190] 5 Ablation Experiment

[0191] To verify the effectiveness of the neural point cloud feature alignment module and the neural point cloud Transformer module in the network model, and to explore the impact of different modules on the quality of synthesized images, ablation experiments were conducted on the Synthetic Blender, Tanks, and Temples datasets. The ablation experiments followed the principle of controlling variables, and were divided into tests with the neural point cloud feature alignment module added, tests with the neural point cloud Transformer module added, and tests with the complete network. The number of iterations was set to 200K, and all parameters were kept consistent throughout the experiments. The baseline network is the experimental result after removing the aforementioned modules. The experimental results are shown in Table 4.

[0192] Table 4 Ablation Experiments: Performance of Different Modules on the Dataset

[0193]

[0194] Bold text is the optimal value for each line.

[0195] 1) Neural point cloud feature alignment module

[0196] As shown in the table, the baseline network had the lowest experimental performance index. However, after adding the neural point cloud feature alignment module alone, the network achieved higher scores on both datasets. This module enhances the adaptability of the neural point cloud to local spatial features by dynamically adjusting the receptive field of the aligned features. This allows the network to focus on relevant regions when processing scene image inputs of different shapes and textures, thereby improving the realism of the synthesized images. Figure 8 A set of visualization results of ablation experiments on the ScanNet dataset is presented. (a1) is the scene image; (a2), (a3), and (a4) are the new perspective scene images obtained by (a1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (a5) is the real perspective scene image of (a1); (b1) is the scene image; (b2), (b3), and (b4) are the new perspective scene images obtained by (b1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (b5) is the real perspective scene image of (b1); (c1) is the scene image; (c2), (c3), and (c4) are the new perspective scene images obtained by (c1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively; (c5) is the real perspective scene image of (c1). Figure 8 As can be seen, unlike the NeRF Synthetic, Tanks and Temples datasets, the ScanNet dataset contains complex scenes with various objects of different shapes, significantly increasing the difficulty of scene reconstruction compared to scenes with a single object. During the formation of neural point clouds, if the alignment features of the point clouds are not accurate enough, noise may be introduced into the neural point cloud features, causing blurring and artifacts, thus affecting the quality of the final result. Furthermore, using this module to form neural point cloud features can also improve the ability of subsequent networks to extract local spatial features.

[0197] (2) Neural Point Cloud Transformer Module

[0198] During encoding, the Neural Point Cloud Transformer module maps the information of neural point clouds into a high-dimensional space to obtain high-frequency spatial features. Using this information, the network can better capture the contextual information around the sampling points and effectively utilize the correlations between neighboring neural point clouds, extracting reliable local spatial features even in sparse point cloud regions. The module improved metrics on both datasets, with a more significant improvement in the synthesized scene images from the poor-quality real-world Tanks and Temples datasets.

[0199] Based on the results of the ablation experiments, the complete network, through the combined effect of each module, showed the best performance in terms of PSNR, SSIM, and LPIP on both datasets. It can effectively utilize the feature information of neural point clouds to synthesize high-quality scene images, verifying the effectiveness of each module in improving network performance.

[0200] Experimental results show that in real-world scene datasets, for the Tanks and Temples dataset containing only a single item, the present invention improves the peak signal-to-noise ratio (PSNR) by 19.2% compared to the NeRF method, and by 6.4% and 3.8% compared to the point cloud input methods Tetra-NeRF and Point-NeRF, respectively. Even in the more complex ScanNet dataset, it improves the PSNR by 34.6% and 2.1% compared to the NeRF method and Point-NeRF, respectively. In conclusion, the present invention can better utilize the local spatial information of point clouds, effectively improving the rendering quality degradation caused by point cloud quality and feature extraction under sparse viewpoint image input. Experimental results verify the effectiveness of the present invention.

[0201] This invention proposes a novel perspective synthesis method that integrates local spatial information. When forming neural point clouds, a feature pyramid network is used to acquire image features that integrate information from different scales of the scene. A neural point cloud feature alignment module adjusts the sampling positions of local features, queries the image feature regions corresponding to each point cloud, and generates neural point clouds. Compared to directly using 2D convolutional networks for downsampling, this method can better focus on relevant regions and capture more useful features, integrating them into the neural point cloud. Before the volume density prediction network, a neural point cloud Transformer module captures the contextual representation of local space. Local neural point cloud features complete their own cues through a self-attention mechanism, effectively handling the sparsity and irregularity of the point cloud and improving the quality of the final synthesized image. Comparative experiments on the Synthetic Blender, Tanks and Temples, and ScanNet datasets show that this invention significantly improves the synthesis results in real-world scenes and weakly textured regions. Although the model of this invention can complete scene reconstruction tasks using point clouds, limitations still exist. Because this invention uses the same volume rendering method as NeRF, the rendering speed is relatively slow; future work will focus on improving the model's real-time performance.

[0202] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A novel perspective synthesis method integrating local spatial information, characterized in that, include: Step 1: Based on the neural point cloud generation network, process multiple sets of images to obtain the final neural point cloud P=(p o ,γ o ,f o ), where each group of images corresponds to a viewpoint, and each group of images contains multiple target images from the same viewpoint, where γ o p represents the confidence level of the point cloud. o f represents the three-dimensional position coordinates. o Features of neural point clouds; Step 1.1: Process multiple sets of images through a deep prediction network to obtain a dense point cloud. The dense point cloud contains multiple point clouds, and each point cloud corresponds to a cloud confidence score and three-dimensional position coordinates. Step 1.2: Using the neural point cloud feature alignment module, process the dense point cloud and multiple sets of images to obtain the final neural point cloud P=(p o ,γ o ,f o ); Step 2: Based on the neural point cloud, construct the neural radiation field network for the final neural point cloud P=(p o ,γ o ,f o The sampling points are processed to obtain the final volume density σ and RGB value c of each sampling point. The sampling points are obtained by sampling multiple times on the line connecting each grid in the initial grid of the view to be synthesized to the center of the camera. The camera is set in a fixed position and has a viewing direction. The initial grid of the view to be synthesized is set in a fixed position. Step 2.1: Using the Neural Point Cloud Transformer module, determine the neural point cloud corresponding to the sampling point, process the neural point cloud corresponding to the sampling point and the 3D coordinates of each sampling point to obtain a new feature vector of the neural point cloud. ,in, Let K be the new feature of the i-th neural point cloud at sampling point x, where K is the number of neural point clouds corresponding to sampling point x. Specifically, for each sampling point, the K nearest neighbor algorithm is used to select the K nearest neural point clouds based on the three-dimensional coordinates of the sampling point. Furthermore, for each of the K neural point clouds, the 3D coordinates of the neural point cloud are subtracted from the 3D coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud. These relative position coordinates are then mapped to a high-dimensional feature space to obtain the mapped relative position coordinates of each neural point cloud. This is achieved through the following formula: ； Where x is the three-dimensional coordinate of the sampling point, p i Let PE(p) be the 3D coordinates of the i-th neural point cloud. i -x) represents the relative position coordinates of the i-th neural point cloud after mapping, and L is the encoding dimension control variable; The mapped relative position coordinates PE(p) i -x) and the features f of the i-th neural point cloud i By cascading, the first intermediate feature f of the neural point cloud is obtained. e Specifically, this is achieved through the following formula: ； in, For feature cascade operations The first intermediate feature f of the neural point cloud e The input is fed into the positional encoding layer to obtain the second intermediate feature PE(f) e Specifically, this is achieved through the following formula: ； The second intermediate feature PE(f) is obtained through MLP. e ) and the first intermediate feature f e The process is performed to obtain the encoded features f of the i-th neural point cloud. i,x Specifically, this is achieved through the following formula: ； All encoded features of neural point clouds constitute the neural point cloud feature sequence X={f i,x |i=1,…,K}; The neural point cloud feature sequence X is input into the attention layer to obtain a new feature vector of the neural point cloud. Specifically, this is achieved through the following formula: ； ； ； ； Where Q is the Query matrix, W q Here, W represents the weight parameters of the Query matrix, K is the Key matrix, and W represents the weight parameters of the Query matrix. k V represents the weight parameters of the Key matrix, V is the Value matrix, and W is the Weight parameter of the Key matrix. v d represents the weight parameters of the Value matrix. k Representing dimension, This represents the output weight matrix, where n represents the total number of attention heads. i This represents the i-th attention head. W represents the weight parameters corresponding to the i-th attention head. q Include , W represents the weight parameters corresponding to the i-th attention head. k Include , W represents the weight parameters corresponding to the i-th attention head. v Include MultiHead(Q,K,V) is a new feature vector of the neural point cloud. ; Step 2.2: Develop new feature vectors for the neural point cloud using a volume density prediction network. The samples are processed to obtain the final volume density σ for each sampling point. Step 2.3: Calculate the RGB value c for each sampling point using the RGB prediction network; Features f encoded from neural point clouds i,x Inverse distance weighting is performed to calculate the weight of each feature vector. The features of the K neural point clouds corresponding to the sampling points are aggregated to obtain the aggregated feature f. x Specifically, this is achieved through the following formula: ； ； The encoded feature PE(d) of the camera view direction d is calculated using the following formula: ； The encoded features PE(d) of the camera view direction d are combined with the aggregated features f x The concatenation is then input into the MLP network to obtain the RGB values ​​c of the sampled points, specifically achieved through the following formula: ； This allows us to obtain the RGB value c for each sampling point; Step 3: Based on the final volume density σ and RGB value c of each sampling point, perform volume rendering on the initial mesh to be synthesized to obtain a synthesized new perspective scene image.

2. The novel perspective synthesis method fusing local spatial information according to claim 1, characterized in that, Step 1.1 specifically includes: By using a multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain a depth map for each viewpoint. Filtering and fusion operations are then performed on all depth maps to obtain a dense point cloud.

3. The novel perspective synthesis method fusing local spatial information according to claim 1, characterized in that, Step 1.2 specifically includes: Step 1.2.1: Using a multi-scale feature fusion method, feature extraction is performed on each target image using the Feature Pyramid Network (FPN) structure to obtain a feature map corresponding to each target image. A deformable convolutional network is then used to process each feature map to obtain an offset feature map. The offset feature map is then meshed to obtain a meshed feature map. Two-dimensional coordinates are set for each grid in the meshed feature map, and the horizontal and vertical coordinates of the meshed feature map are in the range of [-1, 1]. Here, (-1, -1) corresponds to the grid at the upper left corner of the meshed feature map, and (1, 1) corresponds to the grid at the lower right corner of the meshed feature map. Step 1.2.2: Based on the camera pose parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the size of the grid in the gridded feature map. Set two-dimensional coordinates for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1]. Wherein, (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane. Step 1.2.3: Based on the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to that grid. o and three-dimensional position coordinates p o And obtain the image features in the grid, i.e., the neural point cloud features f. o , confidence level γ of point cloud o 3D position coordinates p o and neural point cloud features f o By performing feature concatenation, the final neural point cloud P=(p o ,γ o ,f o ).

4. The novel perspective synthesis method fusing local spatial information according to claim 1, characterized in that, Step 2.2 specifically includes: New feature vectors of neural point clouds The data is input into an MLP network to obtain the volumetric density of each neural point cloud. Specifically, this is achieved through the following formula: ; The final volume density σ of sampling point x is obtained by inverse distance weighting of the confidence score of each neural point cloud, specifically through the following formula: ; ; This allows us to obtain the final volume density σ at each sampling point.

5. The novel perspective synthesis method fusing local spatial information according to claim 1, characterized in that, Step 3 specifically includes: Using a stratified sampling method to divide the interval [t] n ,t f Divide the data into N identical intervals, where t n and t f These are pre-set points on the line connecting the camera to each grid, with a sampling distance randomly selected within each interval, specifically represented by the following formula: ； Among them, t j This represents the sampling distance selected in the j-th interval, and the sampling point position x in the j-th interval is obtained. j Based on the sampling point position x of the j-th interval j Calculate the predicted color for each grid cell in the initial grid. Specifically, this is achieved through the following formula: ； ； ； ； in, Indicates sampling point x j RGB value at position sampling point x j Bulk density at location sampling point x m Bulk density at location, t j-1 This represents the sampling distance selected in the (j-1)th interval. This represents the sampling distance selected in the m-th interval; For each grid cell in the initial grid, predict the color to obtain a synthesized scene image with a new perspective.

6. The novel perspective synthesis method fusing local spatial information according to claim 1, characterized in that, Also includes: Step 4: Calculate the rendering loss between the synthesized new perspective scene image and the real image. Specifically, this is achieved through the following formula: ； Among them, C gt The pixels of the actual image; Introduce a sparsity loss function and calculate the sparsity loss value L. sqarse Specifically, it is expressed by the following formula: ； Where γ represents the confidence level; Based on rendering loss value and sparsity loss value L sqarse The joint loss value L is obtained. opt Specifically, it is expressed by the following formula: ； Where a is a fixed value, ; Based on the joint loss value L opt The neural point cloud generation network and the neural radiation field network were trained multiple times to obtain the trained neural point cloud generation network and the trained neural radiation field network.

Images