New view angle synthesis method fusing local space information

Abstract

The invention provides a new visual angle synthesis method fusing local spatial information, and relates to the technical field of visual angle synthesis, and the method comprises the steps: obtaining image features fusing different scale information of a scene through a feature pyramid network when a neural point cloud is formed, adjusting the sampling position of local features through a neural point cloud feature alignment module, and obtaining a new visual angle. According to the method, the image feature region corresponding to each point cloud is queried, and the neural point cloud is generated, so that compared with a method of directly using a 2D convolutional network to perform down-sampling, the related region can be better concerned, more useful features can be captured, and the features are fused into the neural point cloud. Before a volume density prediction network, context representation of a local space is captured through a neural point cloud Transform module, local neural point cloud features complement own clues through a self-attention mechanism, sparsity and irregularity of point clouds can be effectively processed, and the quality of a finally synthesized picture is improved. According to the method, the synthesis result of the real scene and the weak texture region is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of perspective synthesis, and in particular to a new perspective synthesis method integrating local spatial information. Background Art

[0002] Synthesizing new perspective scene images from a set of 2D images and their associated photographic poses is an important topic in the field of computer vision. Mildenhall et al. (2020) proposed a neural rendering method - neural radiance field (NeRF), which uses a fully connected multi-layer perception (multi-layer perception, MLP) to store real scene information and represents the scene as an implicit neural network. As a novel scene representation method, NeRF has greatly improved the visual effect in rendering tasks. Although the quality of NeRF synthesized images is high, this method sacrifices training time. When reconstructing the scene, each sampling point needs to pass through the entire MLP model. In order to alleviate the problem of the original NeRF reconstructing the scene too slowly, Yu et al. (2021) introduced a hierarchical data structure of PlenOctrees in combination with voxels. This structure uses octrees to avoid oversampling, can reduce sampling points with little contribution, and can achieve real-time rendering of images compared to NeRF. Reiser et al. (2021) proposed a method that does not require The voxel method (Kilo-NeRF) uses thousands of small MLPs to represent scenes, and each small MLP is responsible for describing a part of a scene. This divide-and-conquer strategy increases the computing speed by about 3 orders of magnitude. Fridovich-Keil et al. (2022) completely abandoned neural networks and used sparse voxel grids to store spherical harmonics to represent scenes, and used trilinear interpolation to obtain the RGB values ​​and volume density of the sampling points, which increased the training speed by nearly 2 speed levels. Müller et al. (2022) proposed a multi-resolution hash coding and used CUDA programming to reduce computing time, thereby improving the efficiency of scene reconstruction. It only takes a few seconds to train a high-quality scene.

[0003] On the other hand, many scholars have conducted research on using NeRF to improve the quality of scene reconstruction. To solve the problem of blurring and jagged images, Barron et al. (2021) projected a cone from the camera origin to the "pixel" to replace the light in NeRF, and rendered the pixel by a Gaussian distribution that approximated the conical frustum corresponding to the pixel. Verbin et al. (2022) replaced the parameterized view-dependent outgoing radiance of NeRF with reflected radiance, and constructed the function through a set of spatially varying scene attributes. Compared with the original NeRF, this method can additionally output surface normal, diffuse color, specular weight, and roughness, calculate the specular color based on the viewpoint direction and roughness, and use the diffuse color and specular weight to predict the final color. Wang et al. (2023) modeled surface reflection as a neural network, taking the direction of the incident light and the lighting conditions in the scene as input, and outputting the color and intensity of the surface reflection. By using rendered images with lighting and material information to optimize network parameters during training, the network is able to learn the reflective behavior of complex materials, including the optical properties of various materials such as metals, plastics, and skin.

[0004] Although the quality of new perspective images synthesized by neural radiation fields is high, there are still many problems when only a few views of the scene are available (Chen et al., 2021; Yu et al., 2021; Jain et al., 2021). Many scholars have tried to solve this problem in various ways, including content-based regularization (Jain et al., 2021), patch-based regularization (Niemeyer et al., 2022), using image features (Yu et al., 2021), based on diffusion prior information (Deng et al., 2023; Wynn and Turmukhambetov, 2023), based on deep supervision (Xiao Qiang et al., 2024; Liu Xiaonan et al., 2024), and based on Gaussian rendering (Gaussian Splatting) (Chung et al., 2024). Yu et al. (2021) proposed a pixelNeRF network that allows NeRF to be trained in different scenes. By learning the prior knowledge of the scene, only one or a few pictures are needed to reconstruct the scene. Chen et al. (2021) used a pre-trained convolutional network to extract image features and mapped the image features to the scanning plane of the reference view to construct a cost volume based on three-dimensional voxels, and used a 3D convolutional network to reconstruct the neural code volume, and used MLP to decode the implicit features in the neural code volume into volume density and color values. Xu et al. (2022) used only a single-view image to train the neural radiation field to reconstruct complex visual scenes. It constructed a semi-supervised learning framework and completed the reconstruction of the scene by designing supervision signals based on geometry and semantics. Geometric supervision ensures geometric consistency under multiple views through image deformation and camera pose parameters. Semantic supervision extracts image features through a pre-trained ViT, and then compares them with the overall structure to optimize the semantic quality under unknown viewpoints. Xiao Qiang et al. (2024) used a deep supervision strategy to allocate more sampling points to the surface of the object and introduced light loss under unknown viewpoints, making the final result more refined. Liu Xiaonan et al. (2024) used a depth estimation network to obtain the standard deviation between the estimated depth value and the sparse depth value to supervise the entire network, thereby solving the problem of sparse perspective input. Chung et al. (2024) used a large number of Gaussian renderings to represent 3D scenes and used depth maps to regularize the optimization process of Gaussian rendering, thereby effectively reducing the "floating artifacts" in the results.

[0005] Another way to solve the sparse view problem is to use point cloud information from sensors or photogrammetry. Xu et al. (2022) used the input sparse view image to generate a point cloud, and then input the point cloud features combined with the sampling point information into the neural radiation field for scene reconstruction; Kulhanek and Sattler (2023) meshed the input point cloud and introduced a tetrahedron structure to assist the sampling point interpolation; Zhang et al. (2023) converted the point cloud to the frequency domain through Fourier function for optimization, simplifying the scene representation process; Govindarajan et al. (2024) proposed Lagrangian hashing, a neural radiation field representation method that combines point cloud and hierarchical hash table.

[0006] The point cloud-based neural radiance field method is a new method for reconstructing scenes from sparse viewpoint images. It uses the geometric prior of the point cloud scene to improve the network sampling efficiency and effectively improves the problem of long NeRF training time. However, since this method requires additional input of the point cloud, the quality of the final synthesized new viewpoint image is limited by the accuracy, position distribution, and density of the point cloud. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention aims to propose a new perspective synthesis method that integrates local spatial information, including:

[0008] Step 1: Based on the neural point cloud generation network, multiple groups of images are processed to obtain the final neural point cloud P = (p o ,γ o ,f o ), where each group of images corresponds to a viewing angle, and each group of images contains multiple target images of the same viewing angle, where γ o is the point cloud confidence, p o is the three-dimensional position coordinate, f o is the neural point cloud feature;

[0009] Step 1.1: Process multiple groups of images through a deep prediction network to obtain a dense point cloud, where the dense point cloud contains multiple point clouds, each of which corresponds to a cloud confidence and a three-dimensional position coordinate;

[0010] Step 1.2: Process the dense point cloud and multiple sets of images through the neural point cloud feature alignment module to obtain the final neural point cloud P = (p o ,γ o ,f o );

[0011] Step 2: Based on the neural point cloud, the neural radiation field network is used to calculate the final neural point cloud P = (p o ,γ o ,f o) to obtain the final volume density σ and RGB value c of each sampling point, wherein the sampling point is obtained by performing multiple sampling on the line connecting each grid in the initial grid of the view to be synthesized and the center of the camera, wherein the camera is set at a fixed position, and the camera is set with a view direction, and the initial grid of the view to be synthesized is set at a fixed position;

[0012] Step 3: According to the final volume density σ and RGB value c of each sampling point, the initial mesh to be synthesized is volume rendered to obtain a synthesized new perspective scene image.

[0013] Optionally, step 1.1 specifically includes:

[0014] Through the multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain the depth map of each perspective, and all depth maps are filtered and fused to obtain a dense point cloud.

[0015] Optionally, step 1.2 specifically includes:

[0016] Step 1.2.1: adopt a multi-scale feature fusion method, with the help of a feature pyramid network FPN structure, extract features from each target image to obtain a feature map corresponding to each target image, obtain a shifted feature map for each feature map through a deformable convolutional network, grid the shifted feature map to obtain a gridded feature map, set a two-dimensional coordinate for each grid in the gridded feature map, and the horizontal and vertical coordinates of the gridded feature map are in the range of [-1,1], where (-1,-1) corresponds to the grid in the upper left corner of the gridded feature map, and (1,1) corresponds to the grid in the lower right corner of the gridded feature map;

[0017] Step 1.2.2: According to the camera's posture parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the grid size in the gridded feature map. Set a two-dimensional coordinate for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1], where (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane;

[0018] Step 1.2.3: According to the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to the grid o and the three-dimensional position coordinates p o , and obtain the image features in the grid, that is, the neural point cloud features f o , the confidence of the point cloud γ o, three-dimensional position coordinates p o and neural point cloud features f o Perform feature cascading to obtain the final neural point cloud P = (p o ,γ o ,f o ).

[0019] Optionally, step 2 specifically includes:

[0020] Step 2.1: Determine the neural point cloud corresponding to the sampling point through the neural point cloud Transformer module, process the neural point cloud corresponding to the sampling point and the three-dimensional coordinates of each sampling point, and obtain the new feature vector {f i ' ,x |i=1,...,K}, where f i ' ,x is the new feature of the i-th neural point cloud of sampling point x, and K is the number of neural point clouds corresponding to sampling point x;

[0021] Step 2.2: Through the volume density prediction network, the new feature vector {f i ' ,x |i=1,...,K} to obtain the final volume density σ of each sampling point;

[0022] Step 2.3: Calculate the RGB value c of each sampling point through the RGB prediction network.

[0023] Optionally, step 2.1 specifically includes:

[0024] For each sampling point, the K nearest neighbor algorithm is used to select the K neural point clouds {P i |i=1,...,K}; Then, for each of the K neural point clouds, the three-dimensional coordinates of the neural point cloud are subtracted from the three-dimensional coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud, and the relative position coordinates of each neural point cloud are mapped to the high-dimensional feature space to obtain the relative position coordinates of each neural point cloud after mapping, which is specifically achieved by the following formula:

[0025] PE(p i -x)=(sin(2 0 π(p i -x)),cos(2 0 π(p i -x)),...,sin(2 L-1 π(p i -x))

[0026] ,cos(2 L-1 π(p i-x)));

[0027] Among them, x is the three-dimensional coordinate of the sampling point, p i is the three-dimensional coordinate of the i-th neural point cloud, PE(p i -x) is the relative position coordinate after the i-th neural point cloud is mapped, and L is the encoding dimension control variable;

[0028] The mapped relative position coordinates PE(p i -x) and the feature f of the i-th neural point cloud i Cascade to obtain the first intermediate feature f of the neural point cloud e , which is specifically achieved through the following formula:

[0029] f e =Concat(f i ,PE(p i -x));

[0030] Among them, Concat(·) is the feature cascade operation

[0031] The first intermediate feature f of the neural point cloud e Input to the position encoding layer to obtain the second intermediate feature PE (f e ), which is implemented by the following formula:

[0032] PE(f e )=(sin(2 0 πf e ),cos(2 0 πf e ),...,sin(2 L-1 πf e )

[0033] ,cos(2 L-1 πf e ));

[0034] Through MLP, the second intermediate feature PE(f e ) and the first intermediate feature f e Processing is performed to obtain the encoded feature f of the i-th neural point cloud i,x , which is specifically achieved through the following formula:

[0035] f i,x =MLP(Concat(f e ,PE(f e )));

[0036] All neural point cloud encoded features constitute the neural point cloud feature sequence X = {f i,x |i=1,…,K};

[0037] Input the neural point cloud feature sequence X into the attention layer to obtain the new feature vector {f′ i,x |i=1,...,K}, which is implemented by the following formula:

[0038] (Q,K,V)=X×(W q ,W k ,W v );

[0039]

[0040] MultiHead(Q,K,V)=Concat(head1,...,head n )W O ;

[0041]

[0042] Among them, Q is the Query matrix, W q is the weight parameter of the Query matrix, K is the Key matrix, W k is the weight parameter of the Key matrix, V is the Value matrix, W v is the weight parameter of the Value matrix, d k Represents the dimension, W O represents the output weight matrix, n represents the total number of attention heads, head i represents the i-th attention head, is the weight parameter corresponding to the i-th attention head, W q Include is the weight parameter corresponding to the i-th attention head, W k Include is the weight parameter corresponding to the i-th attention head, W v Include MultiHead(Q,K,V) is the new feature vector of the neural point cloud {f i, ' x |i=1,...,K}.

[0043] Optionally, step 2.2 specifically includes:

[0044] The new feature vector {f′ i,x |i=1,...,K} is input into the MLP network to obtain the volume density {σ i,x |i=1,...,K}, which is implemented by the following formula:

[0045] σ i,x =MLP(f′ i,x );

[0046] The confidence of each neural point cloud is inversely weighted by distance to obtain the final volume density σ of the sampling point x, which is specifically achieved by the following formula:

[0047]

[0048] Then the final volume density σ of each sampling point is obtained.

[0049] Optionally, step 2.3 specifically includes:

[0050] The feature f after encoding the neural point cloud i,x Perform inverse distance weighting, calculate the weight of each feature vector, aggregate the features of the K neural point clouds corresponding to the sampling points, and obtain the aggregated feature f x , which is specifically achieved through the following formula:

[0051]

[0052] Calculate the encoding feature PE(d) of the camera viewing direction d, which is specifically implemented by the following formula:

[0053] PE(d)=(sin(2 0 πd),cos(2 0 πd),...,sin(2 L-1 πd)

[0054] ,cos(2 L-1 πd));

[0055] The encoding feature PE(d) of the camera viewing direction d is combined with the aggregate feature f x Splicing is input into the MLP network to obtain the RGB value c of the sampling point, which is implemented by the following formula:

[0056] c=MLP(Concat(f x ,PE(d)));

[0057] Then the RGB value c of each sampling point is obtained.

[0058] Optionally, step 3 specifically includes:

[0059] The stratified sampling method is used to divide the interval [t n ,t f ] is divided into N equal intervals, where t n and t f It is a pre-set point on the line connecting the camera and each grid, and a sampling distance is randomly selected in each interval, which is specifically expressed by the following formula:

[0060]

[0061] Among them, t j Indicates the sampling distance selected in the jth interval, and obtains the sampling point position x of the jth interval j , according to the sampling point position x of the jth interval j , calculate the predicted color of each grid in the initial grid This is achieved specifically through the following formula:

[0062]

[0063] in, Represents the sampling point x j The RGB value at the position, Sampling point x j The volume density at the location, Sampling point x m The volume density at location, t j-1 represents the sampling distance selected in the j-1th interval, t m represents the sampling distance selected in the mth interval;

[0064] Predict the color of each grid in the initial grid to obtain a synthesized new perspective scene image;

[0065] Optionally, also include:

[0066] Step 4: Calculate the rendering loss value L between the synthesized new perspective scene image and the real image render , which is specifically achieved through the following formula:

[0067]

[0068] Among them, C gt is the pixel of the real image;

[0069] Introduce the sparsity loss function and calculate the sparsity loss value L sqarse , which is specifically expressed by the following formula:

[0070]

[0071] Among them, γ represents the confidence level;

[0072] According to the rendering loss value L render and the sparsity loss value L sqarse , and get the joint loss value L opt , which is specifically expressed by the following formula:

[0073] L opt =L render +aL sparse ;

[0074] Where a is a fixed value, a=2e -3 ;

[0075] According to the joint loss value L opt The neural point cloud generation network and the neural radiation field network are trained multiple times to obtain a trained neural point cloud generation network and a trained neural radiation field network.

[0076] The beneficial effects of adopting the above technical solution are:

[0077] The present invention proposes a new perspective synthesis method that integrates local spatial information. When forming a neural point cloud, a feature pyramid network is used to obtain image features that integrate information of different scales of the scene, and the sampling position of the local features is adjusted through the neural point cloud feature alignment module. The image feature area corresponding to each point cloud is queried and a neural point cloud is generated. Compared with directly using a 2D convolutional network for downsampling, it can better focus on the relevant area and capture more useful features, and integrate them into the neural point cloud. Before the volume density prediction network, the contextual representation of the local space is captured by the neural point cloud Transformer module, and the local neural point cloud features complement their own clues through the self-attention mechanism, which can effectively handle the sparsity and irregularity of the point cloud and improve the quality of the final synthesized picture. The synthesis results of the present invention in real scenes and weak texture areas have been significantly improved. Although the model of the network of the present invention can complete the task of scene reconstruction with the help of point clouds, it still has limitations. Since the present invention adopts the same volume rendering method as NeRF, the rendering speed is slow, and future work will focus on improving the real-time performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Schematic diagram of the process of a new perspective synthesis method integrating local spatial information in an embodiment of the present invention;

[0079] Figure 2 Schematic diagram of the structure of a neural point cloud feature alignment module in an embodiment of the present invention;

[0080] Figure 3 Schematic diagram of the process of a deformable convolution algorithm in an embodiment of the present invention;

[0081] Figure 4 Schematic diagram of the structure of the neural radiation field network of the neural point cloud in an embodiment of the present invention;

[0082] Figure 5: It is a visualization result diagram of the comparative experiment on the Synthetic Blender dataset in the embodiment of the present invention, wherein (a1) is the Ficus scene graph, (a2), (a3), (a4), and (a5) are the new perspective scene images obtained by (a1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (a6) is the real perspective scene image of (a1); (b1) is the Chair scene graph, (b2), (b3), (b4), and (b5) are the new perspective scene images obtained by (b1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (b6) is the real perspective scene image of (a1); ) is the real perspective scene image of (b1); (c1) is the Lego scene graph, (c2), (c3), (c4), and (c5) are the new perspective scene images of (c1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (c6) is the real perspective scene image of (c1); (d1) is the Drums scene graph, (d2), (d3), (d4), and (d5) are the new perspective scene images of (d1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (d6) is the real perspective scene image of (d1);

[0083] Figure 6 : It is a visualization comparison result diagram of the real scene graph in the embodiment of the present invention, wherein (a1) is the Barn scene graph, (a2), (a3), and (a4) are respectively the new perspective scene images of (a1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, and (a5) is the real perspective scene image of (a1); (b1) is the Truck scene graph, (b2), (b3), and (b4) are respectively the new perspective scene images of (b1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, and (b5) is the real perspective scene image of (b1); (c1) is the Caterpilar scene graph, (c2), (c3), and (c4) are respectively the new perspective scene images of (c1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, and (c5) is the real perspective scene image of (c1);

[0084] Figure 7 1 are training performance diagrams of different scenes in the embodiments of the present invention, (a) is a training performance diagram on the Lego scene; (b) is a training performance diagram on the Ficus scene; (c) is a training performance diagram on the Drums scene;

[0085] Figure 8: It is a visualization result diagram of the ablation experiment on the ScanNet dataset in an embodiment of the present invention, wherein (a1) is a scene graph, (a2), (a3), and (a4) are respectively new perspective scene images of (a1) obtained based on the baseline network, the neural point cloud feature alignment network, and the complete network, and (a5) is the real perspective scene image of (a1); (b1) is a scene graph, (b2), (b3), and (b4) are respectively new perspective scene images of (b1) obtained based on the baseline network, the neural point cloud feature alignment network, and the complete network, and (b5) is the real perspective scene image of (b1); (c1) is a scene graph, (c2), (c3), and (c4) are respectively new perspective scene images of (c1) obtained based on the baseline network, the neural point cloud feature alignment network, and the complete network, and (c5) is the real perspective scene image of (c1). DETAILED DESCRIPTION

[0086] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0087] In view of the problems existing in the prior art, the key to solving this problem in the present invention is how to extract reliable local spatial information from limited point cloud features. The present invention improves the correlation between point cloud and local image features, and uses the attention mechanism of Transformer (Vaswani et al., 2017) to capture local neural point cloud context information, and provides a new perspective synthesis method that integrates local spatial information. Specifically: 1) A new point cloud-based neural rendering method is proposed. By integrating the semantic information of the image into the point cloud features, the local neural point cloud features are modeled and information is extracted, which can better synthesize new perspective scenes from sparse image inputs. 2) A neural point cloud feature alignment module is proposed. By adaptively adjusting the convolution kernel size, the point cloud and image features are better aligned, and the expression ability of the neural point cloud for local spatial information is improved, thereby improving the synthesis quality of the entire neural radiation field. 3) Before estimating the volume density, the local spatial information of the neural point cloud around the sampling point is fully utilized, and a Transformer module structure based on the neural point cloud is given, which can effectively improve the local details of the scene rendering picture.

[0088] In order to solve the problem of insufficient point cloud quality and extracted features, the present invention first provides a neural point cloud feature alignment module to align the features of the point cloud and the image matching area, and fuse them to form a neural point cloud, thereby improving the local expression ability of its features. Secondly, a neural point cloud Transformer module is proposed to fuse the contextual information of the local neural point cloud, which can still extract reliable local spatial information when the point cloud quality is poor, effectively enhancing the synthesis quality of the point cloud neural rendering method.

[0089] The present invention provides a new perspective synthesis method integrating local spatial information. Figure 1 The overall network structure mainly consists of two parts: the neural point cloud generation network and the neural radiation field network. The neural point cloud generation network processes the input image and generates a neural point cloud by aligning the point cloud and the extracted image features. In the neural radiation field network part, the RGB value and volume density of the sampling point are predicted by aggregating the neural point cloud features and camera view information near the sampling point, and finally the new view scene picture is synthesized through the volume rendering formula. Specifically, the following steps can be included:

[0090] Step 1: Based on the neural point cloud generation network, multiple groups of images are processed to obtain the final neural point cloud P = (p o ,γ o ,f o ), where each group of images corresponds to a viewing angle, and each group of images contains multiple target images of the same viewing angle, where γ o is the point cloud confidence, p o is the three-dimensional position coordinate, f o is the neural point cloud feature;

[0091] Step 1.1: Process multiple groups of images through a deep prediction network to obtain a dense point cloud, where the dense point cloud contains multiple point clouds, each of which corresponds to a cloud confidence and a three-dimensional position coordinate;

[0092] Through the multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain the depth map of each perspective, and all depth maps are filtered and fused to obtain a dense point cloud.

[0093] In the specific implementation process, the input multi-view image is firstly extracted for features, and the images between different viewpoints are matched for features. The variance values ​​of the feature points at different viewpoints are used to construct the cost volume, and the cost volume is regularized using a 3D convolutional network to generate a probability volume for predicting the depth map. Finally, the depth map of each image is filtered and fused to obtain a dense point cloud. Since the probability volume can represent the probability of the point cloud on the surface of the reconstructed object, the confidence corresponding to the point cloud can be obtained using trilinear interpolation.

[0094] Step 1.2: Process the dense point cloud and multiple sets of images through the neural point cloud feature alignment module to obtain the final neural point cloud P = (p o ,γ o ,f o );

[0095] Combination Figure 2 The schematic diagram of the structure of the neural point cloud feature alignment module in FIG. 1 includes the following steps:

[0096] Step 1.2.1: Using the multi-scale feature fusion method, with the help of the feature pyramid network (FPN) structure, feature extraction is performed on each target image to obtain the feature map corresponding to each target image. This processing method helps to process the detailed information of images of different resolutions, thereby improving the generalization ability of the network model (Lin et al., 2017).

[0097] However, when using a fixed convolution kernel to extract image features, it is difficult to effectively process features of different shapes and textures because the receptive fields of different image locations are the same. In order to better align point cloud and image features and improve the relevance of aligned features, the output features of the FPN network are further processed, and the further processing can be based on convolution.

[0098] For the traditional standard convolution, given a 3×3 convolution kernel R∈{(-1,-1),(-1,0),...,(1,1)}, let x(p) and y(p) represent the features of the input feature map x and the output feature map y at position p, ω n is the convolution kernel weight, and its feature mapping is specifically:

[0099]

[0100] Compared with standard convolution, deformable convolution (Dai et al., 2017; Zhu et al., 2019) can learn additional offsets to spread the sampling positions around, specifically:

[0101] The deformable convolutional network is represented by the following formula:

[0102]

[0103] In the formula, Δp n is a learnable offset to the position, a real number with an unconstrained range. Figure 3 As shown in the figure, the use of standard convolution will incorporate unnecessary image information into the extracted features, while the deformable convolution uses a separate convolution layer to learn the offset when extracting image features. It can adaptively adjust the size of the receptive field according to the different positions and shapes of the object, and focus on sampling the area of ​​interest when aligning the image features in the point cloud.

[0104] Based on the principle of deformable convolution with standard convolution kernel, the present invention selects deformable convolution as the feature offset network, changes the sampling position of the convolution kernel through the feature offset network, dynamically adjusts the receptive field of the aligned features, and makes the aligned features of the point cloud focus on the image area with strong correlation, better adapts to different scene structures and texture features, and enhances the ability of the neural point cloud to express the surrounding spatial information. By learning the additional offset of the sampling position, the feature mapping relationship of the convolution kernel is changed.

[0105] Specifically, a deformable convolutional network is used to obtain a shifted feature map for each of the feature maps, and the shifted feature map is gridded to obtain a gridded feature map, and a two-dimensional coordinate is set for each grid in the gridded feature map, and the abscissa and ordinate of the gridded feature map are in the range of [-1, 1], wherein (-1, -1) corresponds to the grid in the upper left corner of the gridded feature map, and (1, 1) corresponds to the grid in the lower right corner of the gridded feature map;

[0106] Step 1.2.2: According to the camera's posture parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the grid size in the gridded feature map. Set a two-dimensional coordinate for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1], where (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane;

[0107] Step 1.2.3: According to the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to the grid o and the three-dimensional position coordinates p o , and obtain the image features in the grid, that is, the neural point cloud features f o , the confidence of the point cloud γ o , three-dimensional position coordinates p o and neural point cloud features f o Perform feature cascading to obtain the final neural point cloud P = (p o ,γ o ,f o ).

[0108] Step 2: Combine Figure 4 , based on the neural radiation field network of the neural point cloud, the final neural point cloud P = (p o ,γ o ,f o) to obtain the final volume density σ and RGB value c of each sampling point, wherein the sampling point is obtained by performing multiple sampling on the line connecting each grid in the initial grid of the view to be synthesized and the center of the camera, wherein the camera is set at a fixed position, and the camera is set with a view direction, and the initial grid of the view to be synthesized is set at a fixed position;

[0109] When predicting the volume density of the sampling point, the feature f of the neural point cloud is directly used i Input into the MLP network to obtain the volume density σ. This method is not accurate enough and can easily lead to a decrease in the quality of the output. Therefore, processing the neural point cloud features alone is not enough to predict the volume density of complex scenes. In order to solve this problem, before predicting the volume density, the neural point cloud Transformer module is used to better capture the relationship and supplementary information between local neural point clouds, so that the neural point cloud features contain more detail clues, generate a more realistic volume density distribution, and show higher detail fidelity when synthesizing new perspective images. This is achieved specifically through step 2.1.

[0110] Step 2.1: Determine the neural point cloud corresponding to the sampling point through the neural point cloud Transformer module, process the neural point cloud corresponding to the sampling point and the three-dimensional coordinates of each sampling point, and obtain the new feature vector {f i ' ,x |i=1,...,K}, where f i ' ,x is the new feature of the i-th neural point cloud of sampling point x, and K is the number of neural point clouds corresponding to sampling point x;

[0111] For each sampling point, the K nearest neighbor algorithm is used to select the K neural point clouds {P i |i=1,...,K}; Then, for each of the K neural point clouds, the three-dimensional coordinates of the neural point cloud are subtracted from the three-dimensional coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud, and the relative position coordinates of each neural point cloud are mapped to the high-dimensional feature space to obtain the relative position coordinates of each neural point cloud after mapping, which is specifically achieved by the following formula:

[0112] PE(p i -x)=(sin(2 0 π(p i -x)),cos(2 0 π(p i -x)),...,sin(2 L-1 π(p i -x))

[0113] ,cos(2 L-1 π(pi -x)));

[0114] Among them, x is the three-dimensional coordinate of the sampling point, p i is the three-dimensional coordinate of the i-th neural point cloud, PE(p i -x) is the relative position coordinate after the i-th neural point cloud is mapped, and L is the encoding dimension control variable;

[0115] The mapped relative position coordinates PE(p i -x) and the feature f of the i-th neural point cloud i Cascade to obtain the first intermediate feature f of the neural point cloud e , which is specifically achieved through the following formula:

[0116] f e =Concat(f i ,PE(p i -x));

[0117] Among them, Concat(·) is the feature cascade operation

[0118] The first intermediate feature f of the neural point cloud e Input to the position encoding layer, it is further encoded and dimensioned to capture the high-frequency feature information of the neural point cloud, and the second intermediate feature PE (f e ), which is implemented by the following formula:

[0119] PE(f e )=(sin(2 0 πf e ),cos(2 0 πf e ),...,sin(2 L-1 πf e )

[0120] ,cos(2 L-1 πf e ));

[0121] Through MLP, the second intermediate feature PE(f e ) and the first intermediate feature f e Processing is performed to obtain the encoded feature f of the i-th neural point cloud i,x , which can better fit the spatial information of the scene around the sampling point, improve the translation invariance of the network to the point cloud, enhance the network's ability to extract local features of the scene, and strengthen the generalization ability of the entire network model. This is achieved through the following formula:

[0122] f i,x =MLP(Concat(f e ,PE(fe )));

[0123] All neural point cloud encoded features constitute the neural point cloud feature sequence X = {f i,x |i=1,…,K};

[0124] Input the neural point cloud feature sequence X into the attention layer to obtain the new feature vector {f i, ' x |i=1,...,K}, which is implemented by the following formula:

[0125] (Q,K,V)=X×(W q ,W k ,W v );

[0126]

[0127] MultiHead(Q,K,V)=Concat(head1,...,head n )W O ;

[0128]

[0129] Among them, Q is the Query matrix, W q is the weight parameter of the Query matrix, K is the Key matrix, W k is the weight parameter of the Key matrix, V is the Value matrix, W v is the weight parameter of the Value matrix, d k Represents the dimension, W O represents the output weight matrix, n represents the total number of attention heads, head i represents the i-th attention head, is the weight parameter corresponding to the i-th attention head, W q Include is the weight parameter corresponding to the i-th attention head, W k Include is the weight parameter corresponding to the i-th attention head, W v Include MultiHead(Q,K,V) is the new feature vector of the neural point cloud {f i, ' x |i=1,...,K}.

[0130] Step 2.2: Through the volume density prediction network, the new feature vector {f i ' ,x|i=1,...,K} to obtain the final volume density σ of each sampling point;

[0131] The new feature vector {f i ' ,x |i=1,...,K} is input into the MLP network to obtain the volume density {σ i,x |i=1,...,K}, which is implemented by the following formula:

[0132] σ i,x =MLP(f′ i,x );

[0133] The confidence of each neural point cloud is inversely weighted by distance to obtain the final volume density σ of the sampling point x, which is specifically achieved by the following formula:

[0134]

[0135] Then the final volume density σ of each sampling point is obtained.

[0136] Step 2.3: Calculate the RGB value c of each sampling point through the RGB prediction network.

[0137] The feature f after encoding the neural point cloud i,x Perform inverse distance weighting, calculate the weight of each feature vector, aggregate the features of the K neural point clouds corresponding to the sampling points, and obtain the aggregated feature f x , which is specifically achieved through the following formula:

[0138]

[0139] Calculate the encoding feature PE(d) of the camera viewing direction d, which is specifically implemented by the following formula:

[0140] PE(d)=(sin(2 0 πd),cos(2 0 πd),...,sin(2 L-1 πd)

[0141] ,cos(2 L-1 πd));

[0142] Since the RGB value of the sampling point is related to the observation direction, the encoding feature PE(d) of the camera viewing direction d is combined with the aggregate feature f x Splicing is input into the MLP network to obtain the RGB value c of the sampling point, which is implemented by the following formula:

[0143] c=MLP(Concat(f x ,PE(d)));

[0144] Then the RGB value c of each sampling point is obtained.

[0145] The RGB value of the sampling point is predicted through an MLP network. This method only needs to decode the aggregated features while keeping the prediction results sufficiently accurate, which can reduce the computational complexity of the network model.

[0146] The entire process from step 2.1 to step 2.3 can be summarized by the following formula:

[0147] (c,σ)=G(x,d,P1,...P K );

[0148] Where G represents the neural radiation field network.

[0149] Step 3: According to the final volume density σ and RGB value c of each sampling point, the initial mesh to be synthesized is volume rendered to obtain a synthesized new perspective scene image.

[0150] The volume rendering method in NeRF (Kajiya and Von Herzen, 1984) is used to synthesize new perspective images of the scene. Specifically, a layered sampling method is used to convert the interval [t n ,t f ] is divided into N equal intervals, where t n and t f It is a pre-set point on the line connecting the camera and each grid, and a sampling distance is randomly selected in each interval, which is specifically expressed by the following formula:

[0151]

[0152] Among them, t j Indicates the sampling distance selected in the jth interval, and obtains the sampling point position x of the jth interval j , according to the sampling point position x of the jth interval j , calculate the predicted color of each grid in the initial grid This is achieved specifically through the following formula:

[0153]

[0154] in, Represents the sampling point x j The RGB value at the position, Sampling point x j The volume density at the location, Sampling point x m The volume density at location, t j-1represents the sampling distance selected in the j-1th interval, t m represents the sampling distance selected in the mth interval;

[0155] Predict the color of each grid in the initial grid to obtain a synthesized new perspective scene image;

[0156] Step 4: To measure the error between the output image and the true image, use the L2 function as the loss function to calculate the rendering loss value L between the synthesized new perspective scene image and the real image. render , which is specifically achieved through the following formula:

[0157]

[0158] Among them, C gt is the pixel of the real image;

[0159] The confidence γ can represent the probability that the point cloud is on the surface of the object. In order to make the model more biased towards the neural point cloud features of the object surface during training, the sparsity loss function is introduced to calculate the sparsity loss value L sqarse , which is specifically expressed by the following formula:

[0160]

[0161] Among them, γ represents the confidence level;

[0162] This loss function can make the confidence γ of the point cloud approach 0 or 1, and improve the constraints on the point cloud with low contribution. When optimizing each scene, the joint loss function of rendering loss and sparsity loss is used as the overall loss of the network, and a fixed value a=2e is used. -3 Balance the impact of the sparsity loss function on the overall loss. Therefore, according to the rendering loss value L render and the sparsity loss value L sqarse , and get the joint loss value L opt , which is specifically expressed by the following formula:

[0163] L opt =L render +aL sparse ;

[0164] Where a is a fixed value, a=2e -3 ;

[0165] According to the joint loss value L opt The neural point cloud generation network and the neural radiation field network are trained multiple times to obtain a trained neural point cloud generation network and a trained neural radiation field network.

[0166] Based on the above scheme, the following experiments were conducted:

[0167] 1. Experimental environment

[0168] The system is Ubuntu 18.04, the CPU model is Intel Core i9-10900, the memory is 32G, the graphics card model is RTX3090, and the programming language is Python 3.7. The model building and testing are all processed on the Vscode compiler.

[0169] 2. Datasets and Evaluation Metrics

[0170] The peak signal to noise ratio (PSNR), structural similarity index measure (SSIM) and learned perceptual image patch similarity (LPIPS) are used to evaluate the quality of image synthesis (Mildenhall et al., 2020). Among them, the PSNR index focuses more on the pixel value error between the synthetic image and the real image. The larger the value, the smaller the error; the SSIM index is used to evaluate the structural similarity between the two images, ranging from -1 to 1. When the two images are exactly the same, the SSIM value is 1; the LPIPS index is a similarity used to evaluate human perception in images. It aims to compare the visual differences between two images, not just the differences at the pixel level. The smaller the value, the better the visual effect.

[0171] Four commonly used datasets, DTU, NeRF Synthetic, Tanks and Temples, and ScanNet, are used for experiments. DTU is an indoor scene dataset, each scene contains 49 object shooting angles, each angle has 7 brightness levels, and the image resolution is 512×640. NeRF Synthetic is a synthetic dataset containing eight scenes, each scene has 100 training images and 200 test images, these images are completely synthesized by Blender rendering, and the image resolution is 800×800. ScanNet is an indoor scanning dataset, and the experiment will be evaluated on two of the scenes, Scene-101 and Scene-241. The experiment of the present invention is consistent with PointNeRF, sampling 20% ​​of the images, that is, 1463 images of Scene-241 and 1000 images of Scene-101 for training, and the remaining images are used for evaluation. Tanks and Temples is a large outdoor scene dataset, which contains 14 scenes. This experiment will select 5 of them for testing.

[0172] 3. Experimental Details

[0173] First, the model is pre-trained on the DTU dataset, and the training set and test set are divided according to the pixelNeRF and MVSNeRF methods. At this stage, the predicted value of the rendering formula is supervised based on the actual pixel color of the real image, and the entire network is trained end-to-end using only the rendering loss. After pre-training, fine-tuning is performed on various scenes of the Synthetic Blender, Tanks and Temples, and ScanNet datasets. In order to better constrain the point cloud, a joint loss function L is used. opt The network training uses the adaptive learning rate optimization Adam algorithm. By dynamically adjusting the learning rate, the network can more effectively balance the convergence speed and stability during the training process. Set the initial learning rate to 5e -4 The decay rate parameters are set to 0.9 and 0.99, the number of iterations is 200k on the Synthetic Blender, Tanks and Temples datasets, and the number of training rounds is set to 300k on the ScanNet dataset.

[0174] 4. Experimental Results

[0175] The present invention is tested and evaluated on three datasets: NeRFSynthetic, Tanks and Temples, and ScanNet, together with more advanced methods such as Point-NeRF, Tetra-NeRF, and FreqPCR. Table 1 shows the PSNR results of different methods on the synthetic scene NeRF Synthetic dataset. The scene images of this dataset are all synthesized by Blender. Since the lighting environment and clarity of the input images are the results generated under ideal conditions, various point cloud-based models perform well on this dataset. The present invention pays more attention to improving the problem of reduced rendering quality caused by the difficulty in acquiring point clouds in complex environments. The average PSNR of each scene reached 33.45dB, achieving the best result among several methods.

[0176] Table 1 PSNR results of different methods on NeRF Synthetic dataset

[0177]

[0178] Among them, the bold font is the optimal value of each row, the underline is the suboptimal value, and “*” indicates the point cloud-based method.

[0179] Figure 5The visualization results of the comparative experiment on the Synthetic Blender dataset are given, where (a1) is the Ficus scene graph, (a2), (a3), (a4), and (a5) are the new perspective scene images obtained based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (a6) is the real perspective scene image of (a1); (b1) is the Chair scene graph, (b2), (b3), (b4), and (b5) are the new perspective scene images obtained based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (b6) is the real perspective scene image of (a1). (b1) is the real perspective scene image; (c1) is the Lego scene graph, (c2), (c3), (c4), (c5) are the new perspective scene images of (c1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (c6) is the real perspective scene image of (c1); (d1) is the Drums scene graph, (d2), (d3), (d4), (d5) are the new perspective scene images of (d1) based on the NPBG algorithm, the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (d6) is the real perspective scene image of (d1). Figure 5 It can be seen that compared with other similar methods, the present invention can better deal with the problems of holes and noise caused by sparse point clouds. In addition, since the features of neural point clouds come from pictures of different perspectives, combining point clouds with visual images can help reduce the impact of mirror reflections. After using the neural point cloud feature alignment module, the different perspective illumination information contained in each neural point cloud will be more accurate. In the Transformer module of the neural point cloud, the neural point clouds around the sampling points are modeled for correlation, and the contribution rate of each perspective feature to mirror reflection is estimated. Therefore, the rendering results in the mirror reflection area are improved, and the transition in the areas with similar image textures and object edges is more natural.

[0180] Tables 2 and 3 give the PSNR, SSIM, and LPIPS index results of different methods on the real scene Tanks and Temples and ScanNet datasets. Since real scene images are affected by factors such as shooting equipment performance and ambient lighting, the quality of the synthesized point cloud is often poor. Tetra-NeRF triangulates the point cloud to obtain a neural radiation field represented by a tetrahedron, and obtains the features of the sampling points by interpolation, making the image result smoother and reducing the generation of holes, but at the same time, local texture details are also lost. The neural point cloud Transformer module of the present invention does not require additional operations on the point cloud, and significantly improves the local details of the synthesized image based on the locality, position and high-dimensional feature information of the local neural point cloud. As shown in Table 2, the present invention achieved the best results in PSNR and SSIM on the Tanks and Temples dataset, among which LPIPS is only lower than Tetra-NeRF. Therefore, compared with other advanced methods of the same type, the present invention can process real scene images more effectively, has good generalization, and the synthesized image has low distortion and higher similarity with the true image. Figure 6 : The visualization comparison results of the real scene graphs, where (a1) is the Barn scene graph, (a2), (a3), and (a4) are the new perspective scene images of (a1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (a5) is the real perspective scene image of (a1); (b1) is the Truck scene graph, (b2), (b3), and (b4) are the new perspective scene images of (b1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (b5) is the real perspective scene image of (b1); (c1) is the Caterpilar scene graph, (c2), (c3), and (c4) are the new perspective scene images of (c1) based on the Point-NeRF algorithm, the Tetra-NeRF algorithm, and the present invention, respectively, and (c5) is the real perspective scene image of (c1).

[0181] Table 2 Results of different methods on the Tanks and Temples dataset

[0182]

[0183]

[0184] Among them, the bold font is the optimal value of each row, the underline is the suboptimal value, and “*” indicates the point cloud-based method.

[0185] Table 3 Results of different methods on two scenes in ScanNet dataset

[0186]

[0187] Among them, the bold font is the optimal value of each row, the underline is the suboptimal value, and “*” indicates the point cloud-based method.

[0188] ScanNet is an indoor scanning dataset and a classic application scenario of point cloud-based neural radiation fields. The present invention was compared with NeRF, NSVF, and Point-NeRF on this dataset. As shown in Table 3, due to the lack of texture and relative smoothness in most areas of the scene, the traditional neural radiation field NeRF and the voxel-based method NSVF performed poorly on this dataset, while the point cloud-based method can obtain more ideal results in this scenario. The average indicators (PSNR, SSIM, LPIP) of the present invention on the two scenes Scene-101 and Scene-204 are better than Point-NeRF, and the highest scores of 30.95, 0.926, and 0.218 were obtained.

[0189] Combination Figure 7 , where (a) is the training performance graph on the Lego scene; (b) is the training performance graph on the Ficus scene; (c) is the training performance graph on the Drums scene. Figure 7 It can be seen that the visualization results of the PSNR values ​​of the present invention and Point-NeRF on the Lego, Ficus, and Drums scenes in the SyntheticBlender dataset. The results show that when fine-tuning the scene, compared with Point-NeRF, the initialization effect of the present invention in different scenes is better. Under lower iteration rounds, the present invention can also obtain better experimental results. After the same round of iterations, the PSNR value of the synthetic image of the present invention is higher, which shows that when fine-tuning the scene, the present invention can fit different scenes more effectively.

[0190] 5 Ablation Experiment

[0191] In order to verify the effectiveness of the neural point cloud feature alignment module and the neural point cloud Transformer module of the network model and explore the impact of different modules on the quality of the synthesized images, ablation experiments were conducted on the Synthetic Blender and Tanks and Temples datasets. The principle of controlling variables was adopted in the ablation experiment, which was divided into adding the neural point cloud feature alignment module, adding the neural point cloud Transformer module and the test of the complete network. The number of experimental iterations was set to 200K, and the parameters were kept consistent during the experiment. The benchmark network was the experimental result after removing the above modules. The experimental results are shown in Table 4.

[0192] Table 4 Ablation experiment: Effects of different modules on the dataset

[0193]

[0194] Among them, the bold font is the optimal value for each row.

[0195] 1) Neural point cloud feature alignment module

[0196] As can be seen from the table, the experimental results of the baseline network have the lowest index value. After adding the neural point cloud feature alignment module alone, the indexes on both data volumes are higher than the baseline network. This module dynamically adjusts the receptive field of the point cloud aligned features to enhance the adaptability of the neural point cloud to the local spatial features, allowing the network to focus on the relevant areas when processing scene image inputs of different shapes and textures, thereby improving the realism of the synthesized images. Figure 8 A set of visualization results of ablation experiments on the ScanNet dataset are given, where (a1) is a scene graph, (a2), (a3), and (a4) are the new perspective scene images of (a1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively, and (a5) is the real perspective scene image of (a1); (b1) is a scene graph, (b2), (b3), and (b4) are the new perspective scene images of (b1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively, and (b5) is the real perspective scene image of (b1); (c1) is a scene graph, (c2), (c3), and (c4) are the new perspective scene images of (c1) based on the baseline network, the neural point cloud feature alignment network, and the complete network, respectively, and (c5) is the real perspective scene image of (c1). Figure 8 It can be seen that, unlike the NeRF Synthetic and Tanks and Temples datasets, the ScanNet dataset contains complex scenes with objects of various shapes. Compared with scenes with a single object, the difficulty of reconstructing the scene is greatly increased. In the process of forming a neural point cloud, if the alignment features of the point cloud are not accurate enough, noise may be introduced into the neural point cloud features, causing blur and artifacts, thus affecting the quality of the final result. In addition, using this module to form neural point cloud features can also improve the ability of subsequent networks to extract local spatial features.

[0197] (2) Neural Point Cloud Transformer Module

[0198] When encoding, the neural point cloud Transformer module maps the information of the neural point cloud into a high-dimensional space to obtain the high-frequency spatial feature information of the neural point cloud. With this information, the network can better capture the contextual information around the sampling point and effectively utilize the correlation between neighboring neural point clouds. It can also extract reliable local spatial features in areas with sparse point clouds. The indicators on both datasets using this module have been improved. In the real scene Tanks and Temples with poor point cloud quality, the scene images synthesized by this module have been improved more significantly.

[0199] According to the results of the ablation experiment, under the joint action of each module, the complete network showed the best effect in terms of PSNR, SSIM, and LPIP on the two data sets, and can effectively use the feature information of the neural point cloud to synthesize high-quality scene images, verifying the effectiveness of each module in improving network performance.

[0200] Experimental results show that in real scene datasets, for the datasets Tanks and Temples containing only a single object, the peak signal to noise ratio (PSNR) of the present invention is improved by 19.2% compared with the NeRF method, and by 6.4% and 3.8% respectively compared with the methods Tetra-NeRF and Point-NeRF using point cloud input. Even in the ScanNet dataset with more complex scenes, the performance is improved by 34.6% and 2.1% respectively compared with the NeRF method and Point-NeRF. Conclusion The present invention can better utilize the local spatial information of the point cloud, effectively improve the rendering quality degradation caused by the point cloud quality and extracted features under sparse view image input, and the experimental results verify the effectiveness of the present invention.

[0201] The present invention proposes a new perspective synthesis method that integrates local spatial information. When forming a neural point cloud, a feature pyramid network is used to obtain image features that integrate information of different scales of the scene, and the sampling position of the local features is adjusted through the neural point cloud feature alignment module. The image feature area corresponding to each point cloud is queried and a neural point cloud is generated. Compared with directly using a 2D convolutional network for downsampling, it can better focus on the relevant area and capture more useful features, and integrate them into the neural point cloud. Before the volume density prediction network, the context representation of the local space is captured by the neural point cloud Transformer module, and the local neural point cloud features complement their own clues through the self-attention mechanism, which can effectively handle the sparsity and irregularity of the point cloud and improve the quality of the final synthesized image. By conducting comparative experiments on the Synthetic Blender, Tanks and Temples, and ScanNet data sets, the results show that the synthesis results of the present invention in real scenes and weak texture areas have been significantly improved. Although the model of the network of the present invention can complete the task of scene reconstruction with the help of point clouds, it still has limitations. Since the present invention adopts the same volume rendering method as NeRF, the rendering speed is slow, and future work will focus on improving the real-time performance of the model.

[0202] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A new perspective synthesis method integrating local spatial information, characterized in that: include: Step 1: Based on the neural point cloud generation network, multiple groups of images are processed to obtain the final neural point cloud P = (p o ,γ o ,f o ), where each group of images corresponds to a viewing angle, and each group of images contains multiple target images of the same viewing angle, where γ o is the point cloud confidence, p o is the three-dimensional position coordinate, f o is the neural point cloud feature; Step 1.1: Process multiple groups of images through a deep prediction network to obtain a dense point cloud, where the dense point cloud contains multiple point clouds, each of which corresponds to a cloud confidence and a three-dimensional position coordinate; Step 1.2: Process the dense point cloud and multiple sets of images through the neural point cloud feature alignment module to obtain the final neural point cloud P = (p o ,γ o ,f o ); Step 2: Based on the neural point cloud, the neural radiation field network is used to calculate the final neural point cloud P = (p o ,γ o ,f o ) to obtain the final volume density σ and RGB value c of each sampling point, wherein the sampling point is obtained by performing multiple sampling on the line connecting each grid in the initial grid of the view to be synthesized and the center of the camera, wherein the camera is set at a fixed position, and the camera is set with a view direction, and the initial grid of the view to be synthesized is set at a fixed position; Step 3: According to the final volume density σ and RGB value c of each sampling point, the initial mesh to be synthesized is volume rendered to obtain a synthesized new perspective scene image.

2. The new perspective synthesis method for fusing local spatial information according to claim 1, characterized in that: Step 1.1 specifically includes: Through the multi-view stereo vision depth estimation method, the target image in each group of images is processed to obtain the depth map of each perspective, and all depth maps are filtered and fused to obtain a dense point cloud.

3. The new perspective synthesis method integrating local spatial information according to claim 1, characterized in that: Step 1.2 specifically includes: Step 1.2.1: adopt a multi-scale feature fusion method, with the help of a feature pyramid network FPN structure, extract features from each target image to obtain a feature map corresponding to each target image, obtain a shifted feature map for each feature map through a deformable convolutional network, grid the shifted feature map to obtain a gridded feature map, set a two-dimensional coordinate for each grid in the gridded feature map, and the horizontal and vertical coordinates of the gridded feature map are in the range of [-1,1], where (-1,-1) corresponds to the grid in the upper left corner of the gridded feature map, and (1,1) corresponds to the grid in the lower right corner of the gridded feature map; Step 1.2.2: According to the camera's posture parameters, project the three-dimensional position coordinates of the point cloud in the dense point cloud onto the image plane to obtain the projected image plane. The projected image plane is a plane containing multiple grids, and the size of each grid in the projected image plane is the same as the grid size in the gridded feature map. Set a two-dimensional coordinate for each grid in the projected image plane, and the horizontal and vertical coordinates of the projected image plane are in the range of [-1,1], where (-1,-1) corresponds to the grid in the upper left corner of the projected image plane, and (1,1) corresponds to the grid in the lower right corner of the projected image plane; Step 1.2.3: According to the two-dimensional coordinates of each grid, obtain the point cloud confidence γ in the point cloud corresponding to the grid o and the three-dimensional position coordinates p o , and obtain the image features in the grid, that is, the neural point cloud features f o , the confidence of the point cloud γ o , three-dimensional position coordinates p o and neural point cloud features f o Perform feature cascading to obtain the final neural point cloud P = (p o ,γ o ,f o ).

4. The new perspective synthesis method integrating local spatial information according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Determine the neural point cloud corresponding to the sampling point through the neural point cloud Transformer module, process the neural point cloud corresponding to the sampling point and the three-dimensional coordinates of each sampling point, and obtain the new feature vector {f′ i,x |i=1,...,K}, where f′ i,x is the new feature of the i-th neural point cloud of sampling point x, and K is the number of neural point clouds corresponding to sampling point x; Step 2.2: Through the volume density prediction network, the new feature vector {f′ i,x |i=1,...,K} to obtain the final volume density σ of each sampling point; Step 2.3: Calculate the RGB value c of each sampling point through the RGB prediction network.

5. The new perspective synthesis method integrating local spatial information according to claim 4, characterized in that: Step 2.1 specifically includes: For each sampling point, the K nearest neighbor algorithm is used to select the K neural point clouds {P i |i=1,...,K}; Then, for each of the K neural point clouds, the three-dimensional coordinates of the neural point cloud are subtracted from the three-dimensional coordinates of the sampling point to obtain the relative position coordinates of each neural point cloud, and the relative position coordinates of each neural point cloud are mapped to the high-dimensional feature space to obtain the relative position coordinates of each neural point cloud after mapping, which is specifically achieved by the following formula: PE(p i -x)=(sin(2 0 π(p i -x)),cos(2 0 π(p i -x)),...,sin(2 L-1 π(p i -x)),cos(2 L-1 π(p i -x))); Among them, x is the three-dimensional coordinate of the sampling point, p i is the three-dimensional coordinate of the i-th neural point cloud, PE(p i -x) is the relative position coordinate after the i-th neural point cloud is mapped, and L is the encoding dimension control variable; The mapped relative position coordinates PE(p i -x) and the feature f of the i-th neural point cloud i Cascade to obtain the first intermediate feature f of the neural point cloud e , which is specifically achieved through the following formula: f e =Concat(f i ,PE(p i -x)); Among them, Concat(·) is the feature cascade operation The first intermediate feature f of the neural point cloud e Input to the position encoding layer to obtain the second intermediate feature PE (f e ), which is specifically achieved through the following formula: PE(f e )=(sin(2 0 πf e ),cos(2 0 πf e ),...,sin(2 L-1 πf e ),cos(2 L-1 πf e )); Through MLP, the second intermediate feature PE(f e ) and the first intermediate feature f e Processing is performed to obtain the encoded feature f of the i-th neural point cloud i,x , which is specifically achieved through the following formula: f i,x =MLP(Concat(f e ,PE(f e ))); All neural point cloud encoded features constitute the neural point cloud feature sequence X = {f i,x |i=1,…,K}; Input the neural point cloud feature sequence X into the attention layer to obtain the new feature vector {f i, ' x |i=1,...,K}, which is implemented by the following formula: (Q,K,V)=X×(W q ,W k ,W v ); MultiHead(Q,K,V)=Concat(head1,...,head n )W O ; head i =Attention(QW q (i) ,KW k (i) ,VW v (i) ); Among them, Q is the Query matrix, W q is the weight parameter of the Query matrix, K is the Key matrix, W k is the weight parameter of the Key matrix, V is the Value matrix, W v is the weight parameter of the Value matrix, d k Represents the dimension, W O represents the output weight matrix, n represents the total number of attention heads, head i represents the i-th attention head, is the weight parameter corresponding to the i-th attention head, W q Include is the weight parameter corresponding to the i-th attention head, W k Include is the weight parameter corresponding to the i-th attention head, Wv contains MultiHead(Q,K,V) is the new feature vector of the neural point cloud {f′ i,x |i=1,...,K}.

6. The new perspective synthesis method for fusing local spatial information according to claim 4, characterized in that: Step 2.2 specifically includes: The new feature vector {f′ i,x |i=1,...,K} is input into the MLP network to obtain the volume density {σ i,x |i=1,...,K}, which is implemented by the following formula: σ i,x =MLP(f′ i,x ); The confidence of each neural point cloud is inversely weighted by distance to obtain the final volume density σ of the sampling point x, which is specifically achieved by the following formula: Then the final volume density σ of each sampling point is obtained.

7. The new perspective synthesis method integrating local spatial information according to claim 5, characterized in that: Step 2.3 specifically includes: The feature f after encoding the neural point cloud i,x Perform inverse distance weighting, calculate the weight of each feature vector, aggregate the features of the K neural point clouds corresponding to the sampling points, and obtain the aggregated feature f x , which is specifically achieved through the following formula: Calculate the encoding feature PE(d) of the camera viewing direction d, which is specifically implemented by the following formula: PE(d)=(sin(2 0 πd),cos(2 0 πd),...,sin(2 L-1 πd),cos(2 L-1 πd)); The encoding feature PE(d) of the camera viewing direction d is combined with the aggregate feature f x Splicing is input into the MLP network to obtain the RGB value c of the sampling point, which is implemented by the following formula: c=MLP(Concat(f x ,PE(d))); Then the RGB value c of each sampling point is obtained.

8. The new perspective synthesis method for fusing local spatial information according to claim 1, characterized in that: Step 3 specifically includes: The stratified sampling method is used to divide the interval [t n ,t f ] is divided into N equal intervals, where t n and t f It is a pre-set point on the line connecting the camera and each grid, and a sampling distance is randomly selected in each interval, which is specifically expressed by the following formula: Among them, t j Indicates the sampling distance selected in the jth interval, and obtains the sampling point position x of the jth interval j , according to the sampling point position x of the jth interval j , calculate the predicted color of each grid in the initial grid This is achieved specifically through the following formula: in, Represents the sampling point x j The RGB value at the position, Sampling point x j The volume density at the location, Sampling point x m The volume density at location, t j-1 represents the sampling distance selected in the j-1th interval, t m represents the sampling distance selected in the mth interval; The color is predicted for each grid in the initial grid to obtain a synthesized new-view scene image.

9. The new perspective synthesis method integrating local spatial information according to claim 1, characterized in that: Also includes: Step 4: Calculate the rendering loss value L between the synthesized new perspective scene image and the real image render , which is specifically achieved through the following formula: Among them, C gt is the pixel of the real image; Introduce the sparsity loss function and calculate the sparsity loss value L sqarse , which is specifically expressed by the following formula: Among them, γ represents the confidence level; According to the rendering loss value L render and the sparsity loss value L sqarse , and get the joint loss value L opt , which is specifically expressed by the following formula: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> opt <h2 style=";text-align:left;direction:ltr"> =L<h2 style=";text-align:left;direction:ltr"> render <h2 style=";text-align:left;direction:ltr"> +aL<h2 style=";text-align:left;direction:ltr"> sparse <h2 style=";text-align:left;direction:ltr"> ; Where a is a fixed value, a=2e -3 ; According to the joint loss value L opt The neural point cloud generation network and the neural radiation field network are trained multiple times to obtain a trained neural point cloud generation network and a trained neural radiation field network.

Images