A sparse new-view image synthesis method based on multi-scale feature fusion

Abstract

The present invention discloses a sparse new-perspective image synthesis method based on multi-scale feature fusion, which belongs to the field of computer vision and image generation technology, and specifically includes the following steps: S1. Multi-scale reference point generation and feature sampling; S2. Multi-receptive field residual feature extraction; S3. Feature aggregation and image generation based on attention network; S4. Pre-training of new-perspective synthesis model; S5. Fine-tuning of pre-trained model. Different from the existing technology, the present invention proposes a new feature fusion strategy by combining multi-scale features and residual feature extraction methods, and introduces feature aggregation based on attention network for efficient generation of new-perspective images. In addition, the present invention also proposes an adaptive optimization mechanism based on pre-training and transfer learning, which can accelerate the training process in sparse scenes and improve the quality and efficiency of synthesis results. Based on the method of the present invention, the new-perspective image synthesis effect in sparse scenes can be significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics. It is a sparse view neural rendering method based on deep convolutional features and the Vision Transformer (ViT). This method is primarily used to generate high-quality images when view data is limited. It has wide applications in virtual reality (VR), augmented reality (AR), film and television special effects, digital twins, and the metaverse. Background Art

[0002] Novel perspective synthesis is an important problem in computer vision and graphics, aiming to generate high-quality novel perspective images from images with limited viewpoints. Traditional methods typically rely on geometric modeling and explicit representations of the scene, such as multi-view geometry or voxel modeling, which require explicit depth information and scene structure. However, such methods relying on explicit geometry have significant limitations in real-world scenarios, especially when perspectives are sparse or geometric information is missing, resulting in the generated perspectives often lacking realism.

[0003] In recent years, the emergence of implicit representation methods such as neural radiance fields (NeRF) has provided a new paradigm for new perspective synthesis. NeRF uses a neural network to fit the color and density distribution of the scene, and achieves high-quality three-dimensional scene rendering by learning the implicit representation of the radiance field from two-dimensional images without explicit geometric information. This method can generate highly realistic new perspective images and greatly improve the accuracy of new perspective synthesis. However, the application of NeRF still faces two major problems: first, it relies on a large number of perspective images and performs poorly under sparse perspective conditions; second, its computational overhead is extremely high, especially in the inference stage, when multiple sampling along the ray path is required to accumulate color and density information, which is difficult to meet real-time rendering requirements. Reference paper: Mildenhall B, Srinivasan PP, Tancik M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis [J]. Communications of the ACM, 2021, 65(1): 99-106.

[0004] To further improve the performance of novel perspective synthesis, many researchers have introduced advanced network structures and optimization strategies. For example, some improve rendering quality by aggregating features from neighboring perspectives, while others use the Transformer's self-attention mechanism to model features of large-scale scenes. However, these methods still have limitations when using sparse perspective data, mainly reflected in their insufficient ability to capture local and overall spatial information, resulting in rendering results that are not accurate enough in details. Therefore, novel perspective synthesis still faces the challenge of generating high-quality, realistic novel perspective images under sparse perspective data conditions, especially in terms of improving local detail representation, global consistency, and computational efficiency. Research in this field is of great significance for applications such as virtual reality, film and television production, and unmanned driving. Summary of the Invention

[0005] The purpose of the present invention is to provide a sparse new perspective synthesis method based on multi-scale feature fusion, which can greatly improve the processing efficiency and synthesis quality of sparse multi-perspective data, significantly reduce the model's dependence on the number of input perspectives, while retaining important details and global structures in the image, and improve the realism and accuracy of the new perspective generation results.

[0006] A sparse new-view image synthesis method based on multi-scale feature fusion, comprising the following steps:

[0007] S1. Multi-scale reference point generation and feature sampling:

[0008] Generate reference points of different scales according to the resolution of the input image, and use these reference points to sample the corresponding multi-scale feature maps, extract features at each scale and splice them to obtain initial feature information;

[0009] S2. Multi-receptive field residual feature extraction:

[0010] Using multi-receptive field convolution and residual connection mechanisms, the initial feature information of the sample is subjected to depthwise convolution operations with different convolution kernel sizes. Information of different scales is obtained through multiple receptive fields. At the same time, residual connections are added to maintain the original features and fuse the features at each scale, ultimately generating residual features of multiple receptive fields.

[0011] S3. Feature aggregation and image generation based on attention network:

[0012] The fused multi-scale deep features (i.e., residual features) are used as input and processed layer by layer through the self-attention module in the GPNR model. By capturing the global dependencies between different perspectives, multi-view information is gradually aggregated and aligned to complete the deep fusion and expression of features, thereby generating new perspective images in sparse scenes.

[0013] S4. Pre-training of new perspective synthesis model:

[0014] The model is pre-trained on a large-scale, general-purpose dataset. By training on a large number of dense images of different scenes and combining different objects from multiple datasets for multiple iterations, it covers a wide range of possible visual scenes and object types. At the same time, the model weights are optimized using loss function calculation and backpropagation, enabling the model to learn general visual features and high-level semantic features suitable for sparse new perspective synthesis tasks.

[0015] S5. Pre-trained model fine-tuning:

[0016] Further optimization is performed through transfer learning based on pre-trained models. The parameters of the pre-trained model are transferred to the new scene, fine-tuned based on the new scene data under sparse perspective, and the model parameters are updated through backpropagation, thereby accelerating the training and rendering of the new scene, making the model more suitable for specific sparse scenes.

[0017] The resulting model can be used to generate images from new perspectives.

[0018] In step S1, first, multiple scale reference points are generated according to the resolution of the input image. The reference points are key points used to identify the pixel positions in the feature map. The image is divided into a grid of fixed size using an automatic selection method. The center point of the grid is used as the reference point. After the coordinates are normalized, the reference point sets of different scales are merged. Feature sampling is then performed on the corresponding multi-scale feature map (C1, C2, C3). The index position is located in the flattened feature matrix through the normalized coordinates of the reference point, and the eigenvalues ​​are extracted from it and mapped back to the multi-scale structure. These sampled multi-scale features are combined into a matrix.

[0019] The features are spliced ​​and fused into a unified feature representation C. This process is expressed as:

[0020] C=[Sample(C1),Sample(C2),Sample(C3)]

[0021] Here, Sample(·) represents the multi-scale feature sampling based on the reference point, and C contains the fused multi-scale features, providing rich multi-scale information for subsequent processing.

[0022] In step S2, the specific steps of extracting the multi-receptive field residual feature are as follows:

[0023] First, the features C from different scales are linearly transformed through the fully connected layer, different weights and biases are applied to each channel, the input features are re-weighted and combined through matrix operations, the feature relationships between the channels are adjusted and reorganized, and a linearly transformed feature representation (i.e., feature information) is generated:

[0024] FC1(C)=W·C+b

[0025] Among them, C is the multi-scale feature generated in step S1, W and b are parameters obtained after training to help the model better fit the data, and FC1 represents the first fully connected layer.

[0026] Use multi-scale receptive field convolution to perform deep convolution operations on the features after dimensionality reduction. Use convolution kernels of different sizes to convolve the features separately. Split the features in the channel dimension and convolve them one by one. Use convolution kernels of multiple receptive fields to perform independent deep convolution processing on the features of different parts. The specific formula is:

[0027] Conv(x)=[Conv 3×3 (x1),Conv 5×5 (x2),Conv k×k (x3)]

[0028] Here, x1, x2, and x3 are the three parts into which the input feature x is divided along the channel dimension. The channel dimension represents the number of features extracted by the network. Then, different convolution kernel sizes (3×3), (5×5), and (k×k) are used to perform convolution operations on (x1, x2, and x3) to generate features of different scales. Conv represents the convolution operation; k is a custom value. In this invention, the value is 7.

[0029] After the convolution operation at each scale, a residual connection is added to directly add the convolved features to the original input features, keeping the channels of the original input features unchanged, fusing the convolution features and the input features, and merging the two feature representations through the residual connection operation:

[0030] x=[Conv(x),identity(x)]

[0031] Among them, identity(x) means that the input feature x is directly passed without any operation, that is, the input feature is retained.

[0032] The fused multi-scale features are expanded back to the original number of channels through linear projection. The fully connected layer is also used to perform linear transformation on the fused features. The weight matrix and bias are used to weight and add the features to generate a feature representation with the same number of channels as the input features:

[0033] F=FC2([x1,x2,x3])

[0034] Among them, FC2 represents the second fully connected layer, which maps the concatenated multi-scale features back to the target number of channels.

[0035] In step S3, the fused multi-scale features are fed into the GPNR (Generalizable patch-based neural rendering) model for training. The GPNR model receives multi-view fused features as input and uses neural rendering technology to generate a feature representation of the target view. By processing the input features, GPNR can effectively reconstruct and complete missing information in sparse scenes, ultimately outputting high-quality new-view images under sparse conditions, providing complete and consistent feature support for subsequent synthesis and presentation. Reference: Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer. pp. 156–174.

[0036] In step S5, the pre-trained model is used as the initialization of the new model, the weights and bias parameters of the pre-trained model are loaded, and used as the initial parameter settings of the new scene, and then fast fine-tuning is performed on the new scene data under sparse perspective. In the fine-tuning process, the parameters θ0 of the pre-trained model are used as the initial values, and the model is updated by minimizing the loss function.

[0037] Type parameters, Is a weighted loss function, expressed as:

[0038]

[0039] in represents the loss of the pre-trained model, Denotes the loss term for new scene data. By updating the parameters θ via gradient descent with a small number of camera viewpoints, the model can quickly adapt to new scene features, accelerating training and rendering while improving the quality of synthesizing new scene viewpoints.

[0040] Beneficial effects:

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention proposes a sparse new perspective synthesis method based on multi-scale feature fusion, which mainly combines multi-scale feature processing with the residual mechanism to extract depth features. Compared with the existing technology, this method makes full use of multi-scale features and effectively combines the residual structure. The introduction of multi-receptive field convolution and residual connection in multi-scale feature fusion enhances the feature expression ability, enables the model to capture depth information more accurately, and thus improves the rendering quality and global consistency in sparse perspective scenes. In addition, by selectively unfreezing the key layers of the pre-trained model for fine-tuning, the model can quickly adapt to sparse perspective scenes, significantly improving training efficiency and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a network framework diagram of the sparse new perspective synthesis method based on multi-scale feature fusion of the present invention.

[0044] Figure 2 It is a flow chart of the method of the present invention.

[0045] Figure 3 The comparison results of the present invention, variant models and the most advanced methods in different scenarios are shown in Figure 2. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to specific embodiments and accompanying drawings:

[0047] Example 1

[0048] Task Definition

[0049] Assume that a set of sparsely sampled multi-view images is input Each image represents a different camera perspective, and the COLMAP tool is used to accurately calculate the camera pose, including the camera's internal and external reference matrices. The camera pose provides information about the camera's position and orientation in three-dimensional space, determining the perspective of the input image. By adjusting the camera pose, images can be generated from different angles and directions. The goal of this paper is to achieve new perspective reconstruction from sparse input images through neural rendering methods, significantly improving reconstruction quality.

[0050] The present invention is a sparse new perspective synthesis method based on multi-scale feature fusion, which has the following steps:

[0051] (1) Multi-scale reference point generation and feature sampling

[0052] First, according to the resolution H,W of the input image, a multi-scale spatial shape is generated for subsequent segmentation and reshaping of multi-scale features.

[0053] The spatial shape of the multi-scale feature map is defined as:

[0054] (H i ,Wi )=[(H×2,W×2),(H,W),(H / 2,W / 2)]

[0055] These spatial shapes (H i ,W i ) describes the size of each scale feature map, which are high resolution, medium resolution, and low resolution respectively.

[0056] For each feature map scale, a reference point is generated by a uniform grid for spatial positioning of multi-scale features. The reference point coordinate x i and y i Defined as:

[0057] x i =[0.5,1.5,2.5,…,W i -0.5],y i =[0.5,1.5,2.5,…,H i -0.5]

[0058] Generate a two-dimensional grid and expand it into one dimension. The x- and y-coordinates of the reference points are:

[0059]

[0060] Normalize the coordinates of all reference points to obtain the normalized coordinates:

[0061]

[0062] Merge the reference points of all scales to form a unified reference point set. Let N scales =3 corresponds to three scales, and the final reference point set is:

[0063]

[0064] Among them, i=1, 2, 3 correspond to three scales.

[0065] Next, calculate the starting index of each scale to locate the position of each scale feature in the flattened feature. The calculation formula of the starting index is:

[0066] L0=0

[0067]

[0068] Then, using these reference points and the starting index, the flattened features are mapped back to the multi-scale feature map. Given the flattened feature matrix C∈R B×N×C , where B is the batch size, C is the number of channels.

[0069] For each scale i, extract the corresponding feature fragment:

[0070] start i =L i-1

[0071] end i =L i

[0072] C i =C[:,start i :end i ,:]

[0073] Next, multi-scale feature fusion is performed on the feature maps of each scale to generate a multi-scale feature pyramid:

[0074] Pyramid={C0,C1,C2}

[0075] It is then input into the feature extraction network for subsequent multi-receptive field feature extraction and fusion.

[0076] (2) Multi-receptive field residual feature extraction

[0077] A multi-receptive field residual feature extraction module is applied on the multi-scale feature pyramid to further extract and fuse deep features. First, a fully connected layer is applied to perform a linear transformation on the input features:

[0078] X i =FC1(X i )

[0079] For each scale feature (X i )(i=1,2,3), first X i Split equally into two parts in the channel dimension:

[0080]

[0081] Here, X i1 and X i2 Represent the two sub-tensors after segmentation, and the shape of each sub-tensor is That is, it is evenly divided along the channel dimension.

[0082] Then a depthwise separable convolution with different kernel sizes is applied to each sub-tensor:

[0083] Y i1 =Conv 3×3 (X i1 )

[0084] Y i2 =Conv 5×5 (X i2 )

[0085] Among them, Conv k×k represents a depthwise separable convolution with kernel size k×k.

[0086] Then, the convolutional features are concatenated in the channel dimension:

[0087] Y i =[Y i1 ,Y i2 ]

[0088] Note that the concatenation operation here is performed along the channel dimension.

[0089] Apply activation functions (such as GELU) and batch normalization operations. First, apply the GELU activation function to the concatenated features to enhance the nonlinear expression capability, and then perform batch normalization:

[0090] Z i =BN i sGELU(Y i )x

[0091] Among them, the activation function GELU is defined as:

[0092]

[0093] The mathematical representation of batch normalization is:

[0094]

[0095] Among them, μ i represents the average value of the i-th scale feature, σ i represents the variance of the i-th scale feature; γ i and β i is the training parameter, the initial value γ i =1,β i =0.

[0096] Then, the processed features are concatenated with the initial input features in the channel dimension (residual connection):

[0097] Z′ i =[Z i ,X i ]

[0098] Finally, the features of all scales are concatenated along the channel dimension to form the final feature representation:

[0099] Z=[Z′1,Z′2]

[0100] At this time, Z∈R B×H×W×2C, where B is the batch size, H and W are the height and width of the feature map, and C is the number of channels. The last layer is the second fully connected layer, which maps the features back to the output dimension:

[0101] Z out =FC2(Z)

[0102] The role of the second fully connected layer is to map the high-dimensional features of the network to the required output dimension for subsequent feature aggregation tasks.

[0103] (3) Transformer-based multi-view feature aggregation model

[0104] In novel perspective synthesis, the GPNR model achieves novel perspective image synthesis under sparse perspective conditions through Transformer-based multi-view feature aggregation. Its core idea is to use a multi-head self-attention mechanism to capture the global relationship between different perspectives and gradually aggregate depth and feature information by combining geometric consistency modeling. Reference: Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer, pp. 156–174.

[0105] The present invention inputs the obtained deep features into the GPNR network to perform multi-view feature aggregation and new view image generation:

[0106] F output =GPNR(Z out )

[0107] Among them F output It is a feature representation used to predict the target perspective, which can be used to further generate images from new perspectives.

[0108] (4) Generate target image

[0109] The target feature representation output by GPNR is input into a multi-layer perceptron (MLP) to predict the color of the target ray:

[0110]

[0111] (5) Loss function design

[0112] To improve model performance, GPNR designs fine loss and regularization loss based on color supervision to enhance the ability to fit the color and texture of the target scene while suppressing the risk of overfitting:

[0113] Fine loss It is used to measure the color difference between the model predicted image and the real image, which is defined as:

[0114]

[0115] in, is the predicted pixel color value, p i is the true pixel color value, and N is the total number of pixels in the image, which is the same variable as N above.

[0116] Regularization loss By limiting the size of the model weights, the risk of overfitting is reduced, which is defined as:

[0117]

[0118] Wherein, λ is the regularization coefficient. In the present invention, λ is set to 0.01, and w represents the trainable weight of the model.

[0119] The final total loss function is:

[0120]

[0121] Here, α is a weight factor, which is set to 100 in the present invention to balance the relative importance of the fine loss and the regularization loss.

[0122] This loss design effectively improves the model's quality of synthesizing new perspective images under sparse perspective conditions, while ensuring the model's training stability and generalization ability.

[0123] (7) Pre-trained model fine-tuning

[0124] Using a pre-trained GPNR model as a foundation, a general-purpose model pre-trained on a large dataset, the present invention then fine-tunes the model for the target scene using a small number of images with known viewpoints and their corresponding geometric information. During the fine-tuning process, the weights of some high-level feature aggregation modules and geometric consistency modeling in the pre-trained model are frozen, and only the parameters of the low-level feature extraction modules are trained to adapt to the color and texture characteristics of the target scene:

[0125]

[0126] in represents the loss of the pre-trained model, Denotes the loss term for new scene data. By updating the parameters θ via gradient descent with a small number of camera viewpoints, the model can quickly adapt to new scene features, accelerating training and rendering while improving the quality of synthesizing new scene viewpoints.

[0127] Example 2

[0128] like Figures 1-2 As shown, the present invention provides a sparse new perspective synthesis method based on multi-scale feature fusion, comprising the following steps:

[0129] Step 1: Pre-training of new view synthesis model

[0130] Prepare and preprocess diverse image datasets, train the model on large-scale, general datasets, use loss functions to measure errors and update weights through backpropagation, repeat training until convergence, and combine iterative training of multiple datasets to improve generalization capabilities. Finally, save the optimized model weights to provide common features and initial parameters for sparse new perspective synthesis tasks.

[0131] Step 2: Multi-scale reference point generation and feature sampling

[0132] First, for the input sparse view image, multi-scale reference points are generated based on its resolution and feature sampling is performed. During this process, multi-scale spatial shapes, such as (H×2, W×2), (H, W), and (H / 2, W / 2), are calculated to provide a resolution basis for feature maps of different scales. A two-dimensional grid of reference points is generated using a grid generation method. After normalization, the reference point coordinates are mapped to the range [0, 1] to form spatial information guiding sampling. Subsequently, based on these reference points, feature sampling is performed on feature maps of different resolutions to extract multi-scale features at high, medium, and low resolutions. These features are fused into a unified feature representation through a splicing operation, providing a foundation for subsequent processing.

[0133] Step 3: Multi-receptive field residual feature extraction

[0134] During the multi-scale feature fusion stage, the present invention designs a feature processing module based on multi-receptive field convolution and residual mechanisms. The multi-receptive field convolution module extracts local and global contextual information for high-, medium-, and low-resolution features using deep convolution operations with different convolution kernels, such as 3×3, 5×5, and k×k. After each convolution operation, a residual connection is added to maintain the information integrity of the input features and fuse them with the convolved features. The fused multi-scale features are further processed through activation functions and dimensionality reduction operations, and finally reshaped into a unified feature representation with richer expressive power, laying a solid foundation for the generation of new perspective images.

[0135] Step 4: Transformer model training based on multi-view feature aggregation

[0136] In the training phase based on multi-view feature aggregation, the present invention introduces the GPNR model, with Transformer as the core module, to gradually achieve efficient aggregation and fusion of multi-view information. The fused multi-scale features are first input into the self-attention mechanism to extract the global correlation between features from different perspectives. Subsequently, through geometric modeling, the multi-view features are further integrated in the epipolar dimension to capture the depth consistency information of the ray direction. In the stage of fusing the reference view features, the present invention adopts a weighted summation mechanism to generate the final target ray features according to the attention weight distribution for feature reconstruction of the target perspective.

[0137] Step 5: Fine-tune the pre-trained model:

[0138] During the fine-tuning phase of the pre-trained model, the present invention uses an optimization method based on transfer learning to accelerate the training and rendering of new scenes. During the pre-training phase, the GPNR model is trained using large-scale multi-view image data to learn the fusion and reconstruction capabilities of multi-view features. For specific sparse view scenes, the model parameters are further updated through fine-tuning to make it more suitable for specific scene data. In the fine-tuning process, two loss functions are used: fine prediction loss and fine prediction loss. and regularization loss Avoid model overfitting. In practical applications, fewer parameters are fine-tuned, which greatly increases training speed and significantly improves the quality of generated results.

[0139] Step 6: Output the rendered image with the new perspective:

[0140] Finally, the target ray features generated by the above processing are input into the rendering module to generate a new perspective image. Specifically, the model reconstructs the features under sparse perspective conditions based on the color prediction of the target ray, forms the pixel values ​​under the target perspective, and combines all rays to generate a complete new perspective image. The present invention achieves high-quality new perspective image generation under sparse perspective conditions by combining multi-scale feature fusion with multi-perspective information modeling, significantly improving the generalization ability and generation efficiency of the model, while reducing the density requirement of perspective sampling, and has a wide range of practical application value.

[0141] Step 1 only needs to be executed once in this embodiment, while steps 2 to 4 are an iterative process. Finally, the new perspective image is obtained through step 5. The iterative stopping condition is to run for 20 hours on the NVIDIA GeForce RTX 2070 GPU or the overall loss function reaches 3×10 within 1000 iterations. -3 within the threshold.

[0142] Experimental results

[0143] 1. Dataset

[0144] This experiment uses the Co3D dataset, which contains a rich set of categories and objects and is therefore widely used for multi-view image generation and rendering tasks. In the experimental design, we selected a variety of scenes, each using six images taken from different angles for training, rendering, and evaluation. Each rendering generates three images from different perspectives, and then compares these renderings with the real images. See the reference: Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C., 2017. Transformation-grounded image generation network for novel 3D view synthesis, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3500–3509.

[0145] To ensure the accuracy of image generation, we used the COLMAP tool to generate the camera pose for the images. The camera pose determines the angle from which the scene is viewed and directly affects the viewing angle and quality of the rendered image. By adjusting the camera pose, we generated images from multiple viewpoints, thereby testing the model's generalization and rendering performance in multi-view generation.

[0146] 2. Experimental Setup

[0147] The experiment was developed based on the JAX framework and trained using the Adam optimizer. The initial learning rate was set to η0 = 2 × 10 -5 The training time for each scene is approximately 20 hours to ensure optimal performance under limited resource conditions. We run the experiments on an NVIDIA GeForce RTX 2070 GPU with a batch size of 8. In each scene, six images with uniform viewpoints are selected as input, and three images with new viewpoints are generated.

[0148] This experiment selected Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as the main evaluation indicators to measure the quality of generated images. These indicators can comprehensively reflect the accuracy, structural similarity, and visual effect between the generated images and the real images.

[0149] 3. Performance comparison

[0150] We compare our baseline model with the full model and recent state-of-the-art methods as follows:

[0151] GPNR[Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer. pp. 156–174.]: It is a neural radiance field model based on local patch features. It uses a patch-based neural network rendering method to decompose complex three-dimensional scenes into multiple small blocks for rendering.

[0152] LFNR[Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022b. Light field neural rendering, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 8269–8279.]: It is a neural rendering model based on light field representation for sparse perspectives. By introducing light field technology to directly model the relationship between perspective and light, it avoids the complex volume rendering calculations in traditional NeRF, thereby achieving more efficient rendering speed and better new perspective synthesis effects.

[0153] WAH [Bao, Y., Li, Y., Huo, J., Ding, T., Liang, X., Li, W., Gao, Y., 2023. Where and how: Mitigating confusion in neural radiance fields from sparse inputs, in: Proceedings of the 31st ACM International Conference on Multimedia, pp. 2180–2188.] is a window-aware hashing acceleration method for neural radiance fields, focusing on optimizing training efficiency and inference performance in sparsely sampled scenarios. By introducing a window-aware hashing mechanism, WAH effectively captures local features while avoiding the resource waste of traditional global hashing methods under sparse viewport conditions, thereby achieving efficient scene modeling and fast new viewport rendering.

[0154] (1) Comparison results of the method proposed in this paper with other methods in different scenarios

[0155] Table 1 Performance comparison of different methods in different scenarios

[0156]

[0157]

[0158] By comparing the characteristics and goals of all methods, in all scenarios, all indicators of the method of the present invention are better than other comparison methods, and the rendered images are more realistic and clear. As shown in Table 1, the method of the present invention has achieved significant performance improvement in various scenarios, especially in the PSNR and SSIM indicators. Compared with other methods (such as LFNR, WAH, GPNR), the method of the present invention has an average improvement of 4-14 values ​​in PSNR value, and SSIM has also reached a higher value. At the same time, the LPIPS value is significantly reduced, indicating that it has obvious advantages in detail preservation, structure restoration and visual perception quality of image rendering. These results demonstrate the excellent performance of the method of the present invention in image rendering tasks, and can achieve higher image quality and consistency in a variety of scenarios.

[0159] like Figure 3 As shown, in the three scenes of plants, books and bears, the method of the present invention is compared with GPNR, LFNR and WAH, and different rendered images are obtained. In the plant scene, the method of the present invention can clearly show the texture and edges of the leaves, while the other methods appear relatively blurry, especially LFNR and WAH, which can hardly see the details of the leaves and the real texture of the background. In the book scene, the method of the present invention accurately restores the text on the cover and the details of the pages, while the other methods have defects in detail level and color saturation. GPNR does not render details and colors well, and the text is not clear enough. LFNR is blurry and severely distorted, and WAH's rendering loses almost all details and is difficult to identify. In the bear scene, the method of the present invention shows the real texture of the hair and accurately restores the complete photo frame in the background, and the edges of the photo frame and the content inside can be clearly seen. These experimental results show that the method of the present invention has significant advantages in improving image rendering quality, especially in the case of sparse input number, it can better maintain global information and improve the quality of local details.

[0160] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A sparse new-view image synthesis method based on multi-scale feature fusion, characterized in that: The following steps are involved: S1. Multi-scale reference point generation and feature sampling: Generate reference points of different scales according to the resolution of the input image, and use these reference points to sample the corresponding multi-scale feature maps, extract features at each scale and splice them to obtain initial feature information; S2. Multi-receptive field residual feature extraction: Using multi-receptive field convolution and residual connection mechanisms, the initial feature information of the sample is subjected to depthwise convolution operations with different convolution kernel sizes. Information of different scales is obtained through multiple receptive fields. At the same time, residual connections are added to maintain the original features and fuse the features at each scale, ultimately generating residual features of multiple receptive fields. S3. Feature aggregation and image generation based on attention network: The fused multi-scale deep features are used as input and processed layer by layer through the self-attention module in the GPNR model. By capturing the global dependencies between different perspectives, multi-view information is gradually aggregated and aligned to complete the deep fusion and expression of features, thereby generating new perspective images in sparse scenes. S4. Pre-training of new perspective synthesis model: The model is pre-trained on a large-scale, general-purpose dataset. By training on a large number of dense images of different scenes and combining different objects from multiple datasets for multiple iterations, it covers a wide range of possible visual scenes and object types. At the same time, the model weights are optimized using loss function calculation and backpropagation, enabling the model to learn general visual features and high-level semantic features suitable for sparse new perspective synthesis tasks. S5. Pre-trained model fine-tuning: Further optimization is performed through transfer learning based on pre-trained models. The parameters of the pre-trained model are transferred to the new scene, fine-tuned based on the new scene data under sparse perspective, and the model parameters are updated through backpropagation, thereby accelerating the training and rendering of the new scene, making the model more suitable for specific sparse scenes. The resulting model can be used to generate images from new perspectives.

2. The sparse new perspective synthesis method based on multi-scale feature fusion according to claim 1 is characterized in that: In step S1, reference points of multiple scales are first generated based on the resolution of the input image. Reference points are key points used to identify pixel positions in the feature map. An automatic selection method is used to divide the image into a fixed-size grid. The center point of the grid is used as the reference point. After normalizing the coordinates, the reference point sets of different scales are merged. Feature sampling is then performed on the corresponding multi-scale feature map (C1, C2, C3). The normalized coordinates of the reference point are used to locate the index position in the flattened feature matrix, from which the eigenvalues ​​are extracted and mapped back to the multi-scale structure. These sampled multi-scale features are spliced ​​and fused into a unified feature representation C. This process is expressed as: C=[Sample(C1),Sample(C2),Sample(C3)] Here, Sample(·) represents the multi-scale feature sampling based on the reference point, and C contains the fused multi-scale features, providing rich multi-scale information for subsequent processing.

3. The sparse new perspective synthesis method based on multi-scale feature fusion according to claim 1, characterized in that: In step S2, the specific steps of extracting the multi-receptive field residual feature are as follows: First, the features C from different scales are linearly transformed through the fully connected layer, different weights and biases are applied to each channel, and the input features are re-weighted and combined through matrix operations. The feature relationships between the channels are adjusted and reorganized to generate a linearly transformed feature representation: FC1(C)=W·C+b Where C is the multi-scale feature generated in step S1, W and b are parameters obtained through training to help the model better fit the data, and FC1 represents the first fully connected layer; Use multi-scale receptive field convolution to perform deep convolution operations on the features after dimensionality reduction. Use convolution kernels of different sizes to convolve the features separately. After dividing the features in the channel dimension, convolve them one by one. Use convolution kernels of multiple receptive fields to perform independent deep convolution processing on the features of different parts. The specific formula is: Conv(x)=[Conv 3×3 (x1),Conv 5×5 (x2),Conv k×k (x3)] Here, x1, x2, and x3 are the three parts into which the input feature x is divided along the channel dimension. The channel dimension represents the number of features extracted by the network. Then, different convolution kernel sizes (3×3), (5×5), and (k×k) are used to perform convolution operations on (x1, x2, and x3) to generate features of different scales. Conv represents the convolution operation. After the convolution operation at each scale, a residual connection is added to directly add the convolved features to the original input features, keeping the channels of the original input features unchanged, fusing the convolution features and the input features, and merging the two feature representations through the residual connection operation: x=[Conv(x),identity(x)] Among them, identity(x) means that the input feature x is directly passed without any operation, that is, the input feature is retained. The fused multi-scale features are expanded back to the original number of channels through linear projection. The fully connected layer is also used to perform linear transformation on the fused features. The weight matrix and bias are used to weight and add the features to generate a feature representation with the same number of channels as the input features: F=FC2([x1,x2,x3]) Among them, FC2 represents the second fully connected layer, which maps the concatenated multi-scale features back to the target number of channels.

4. The sparse new perspective synthesis method according to claim 1, characterized in that: In step S3, the fused multi-scale features are input into the GPNR (Generalizable patch-based neural rendering) model for training. The GPNR model receives multi-view fusion features as input and uses neural rendering technology to generate feature expressions of the target view. By processing the input features, the GPNR can effectively reconstruct and complete the missing information in sparse scenes, and finally output high-quality new view images under sparse conditions, providing complete and consistent feature support for subsequent synthesis and presentation.

5. The sparse new perspective synthesis method according to claim 1, characterized in that: In step S5, the pre-trained model is used as the initialization of the new model, the weights and bias parameters of the pre-trained model are loaded, and used as the initial parameter settings of the new scene, and then fast fine-tuning is performed on the new scene data under sparse perspective. During the fine-tuning process, the parameters θ0 of the pre-trained model are used as the initial values, and the model parameters are updated by minimizing the loss function. Is a weighted loss function, expressed as: in represents the loss of the pre-trained model, Denotes the loss term for new scene data. By updating the parameters θ via gradient descent with a small number of camera viewpoints, the model can quickly adapt to new scene features, accelerating training and rendering while improving the quality of synthesizing new scene viewpoints.

Images