Sparse new view angle image synthesis method based on multi-scale feature fusion

Abstract

The invention discloses a sparse new view angle image synthesis method based on multi-scale feature fusion, and belongs to the technical field of computer vision and image generation, and the method specifically comprises the following steps: S1, multi-scale reference point generation and feature sampling; s2, multi-receptive field residual feature extraction; s3, feature aggregation and image generation based on the attention network; s4, pre-training a new view angle synthesis model; and S5, finely adjusting the pre-training model. Compared with the prior art, the method has the advantages that a new feature fusion strategy is provided by combining a multi-scale feature extraction method and a residual feature extraction method, and meanwhile, feature aggregation based on the attention network is introduced for efficient generation of a new view angle image. In addition, the invention further provides a self-adaptive optimization mechanism based on pre-training and transfer learning, so that the training process in a sparse scene can be accelerated, and the quality and efficiency of a synthesis result are improved. Based on the method provided by the invention, the new view angle image synthesis effect in a sparse scene can be obviously improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and computer graphics, and is a sparse view neural rendering method based on deep convolutional features and vision transformer (ViT). The method is mainly used to generate quality images when the view data is limited, and is widely used in virtual reality (VR), augmented reality (AR), film and television special effects, digital twins and metaverse. Background Art

[0002] New view synthesis is an important problem in computer vision and graphics, which aims to generate high-quality new view images from limited view images. Traditional methods usually rely on geometric modeling and explicit representation of the scene, such as multi-view geometry or voxel modeling, which require explicit depth information and scene structure. However, this method that relies on explicit geometry has great limitations in real scenes, especially when the view is sparse or the geometric information is missing, the generated view often lacks realism.

[0003] In recent years, the emergence of implicit representation methods such as neural radiance fields (NeRF) has provided a new paradigm for new view synthesis. NeRF achieves high-quality 3D scene rendering by fitting the color and density distribution of the scene using a neural network, and by learning the implicit representation of the radiance field from a 2D image without explicit geometric information. This method can generate highly realistic new view images, greatly improving the accuracy of new view synthesis. However, the application of NeRF still faces two major problems: first, it relies on a large number of view images and performs poorly under sparse view conditions; second, its computational overhead is extremely high, especially in the inference stage, when multiple sampling along the ray path is required to accumulate color and density information, which is difficult to meet the requirements of real-time rendering. Reference paper: Mildenhall B, Srinivasan PP, Tancik M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis [J]. Communications of the ACM, 2021, 65 (1): 99-106.

[0004] In order to further improve the performance of new perspective synthesis, many researchers have introduced advanced network structures and optimization strategies. For example, some improve the rendering quality by aggregating the features of neighboring perspectives, and some use the self-attention mechanism of Transformer to model the features of large-scale scenes. However, these methods still have limitations under sparse perspective data, which is mainly reflected in the insufficient ability to capture local and overall spatial information, resulting in inaccurate rendering results in details. Therefore, new perspective synthesis still faces the challenge of how to generate high-quality and realistic new perspective images under sparse perspective data conditions, especially in improving local detail performance, global consistency and computational efficiency. Research in this field is of great significance for applications such as virtual reality, film and television production, and unmanned driving. Summary of the invention

[0005] The purpose of the present invention is to provide a sparse new perspective synthesis method based on multi-scale feature fusion, which can greatly improve the processing efficiency and synthesis quality of sparse multi-perspective data, significantly reduce the model's dependence on the number of input perspectives, while retaining important details and global structures in the image, thereby improving the realism and accuracy of the new perspective generation results.

[0006] A sparse new-view image synthesis method based on multi-scale feature fusion, comprising the following steps:

[0007] S1. Multi-scale reference point generation and feature sampling:

[0008] Generate reference points of different scales according to the resolution of the input image, and use these reference points for sampling on the corresponding multi-scale feature map, extract features at each scale and splice them to obtain initial feature information;

[0009] S2. Multi-receptive field residual feature extraction:

[0010] Using multi-receptive field convolution and residual connection mechanism, deep convolution operations with different convolution kernel sizes are performed on the sampled initial feature information, and information of different scales is obtained through multiple receptive fields. At the same time, residual connections are added to maintain the original features and fuse the features at each scale, and finally residual features of multiple receptive fields are generated;

[0011] S3. Feature aggregation and image generation based on attention network:

[0012] The fused multi-scale deep features (i.e., residual features) are used as input and processed layer by layer through the self-attention module in the GPNR model. The multi-view information is gradually aggregated and aligned by capturing the global dependencies between different viewpoints to complete the deep fusion and expression of features, thereby generating new viewpoint images in sparse scenes.

[0013] S4. Pre-training of new perspective synthesis model:

[0014] The model is pre-trained on a large-scale, general-purpose dataset. By training a large number of dense images of different scenes and combining different objects in multiple datasets for multiple iterations of training, it covers various possible visual scenes and object types. At the same time, the model weights are optimized by loss function calculation and back propagation, so that the model can learn general visual features and high-level semantic features suitable for sparse new perspective synthesis tasks.

[0015] S5. Pre-trained model fine-tuning:

[0016] Further optimization is performed through transfer learning based on pre-trained models, the parameters of the pre-trained models are transferred to new scenes, and fine-tuned based on the new scene data under sparse perspectives. The model parameters are updated through back propagation, thereby accelerating the training and rendering of new scenes, making the model more suitable for specific sparse scenes.

[0017] The resulting model can be used to generate images from new perspectives.

[0018] In step S1, firstly, reference points of multiple scales are generated according to the resolution of the input image. The reference points are key points used to identify the pixel positions in the feature map. The image is divided into a grid of fixed size by an automatic selection method. The center point of the grid is used as the reference point. After the coordinates are normalized, the reference point sets of different scales are merged. Then, in the corresponding multi-scale feature map (C 1 ,C 2 ,C 3 ), locate the index position in the flattened feature matrix through the normalized coordinates of the reference point, extract the eigenvalues ​​from it and map it back to the multi-scale structure.

[0019] The features are concatenated and fused into a unified feature representation C. This process is expressed as:

[0020] C=[Sample(C 1 ),Sample(C 2 ),Sample(C 3 )]

[0021] Where Sample(·) represents the multi-scale feature sampling based on the reference point, and C contains the fused multi-scale features, providing rich multi-scale information for subsequent processing.

[0022] In step S2, the specific steps of extracting the multi-receptive field residual features are as follows:

[0023] First, the features C from different scales are linearly transformed through the fully connected layer, different weights and biases are applied to each channel, the input features are re-weighted and combined through matrix operations, the feature relationships between channels are adjusted and reorganized, and a linearly transformed feature representation (i.e., feature information) is generated:

[0024] FC 1 (C) = W·C + b

[0025] Among them, C is the multi-scale feature generated in step S1, W and b are parameters obtained after training to help the model better fit the data, and FC 1 Represents the first fully connected layer (Fully Connected Layer).

[0026] Use multi-scale receptive field convolution to perform deep convolution operations on the features after dimensionality reduction. Use convolution kernels of different sizes to convolve the features separately. Split the features in the channel dimension and convolve them one by one. Use convolution kernels of multiple receptive fields to perform independent deep convolution processing on the features of different parts. The specific formula is:

[0027] Conv(x)=[Conv 3×3 (x 1 ),Conv 5×5 (x 2 ),Conv k×k (x 3 )]

[0028] Among them, x 1 ,x 2 ,x 3 The input feature x is divided into three parts in the channel dimension. The channel dimension represents the number of features extracted by the network. Then, different convolution kernel sizes (3×3), (5×5), and (k×k) are used to convolution (x 1 ,x 2 ,x 3 ) performs convolution operation to generate features of different scales. Conv represents convolution operation; k is a custom value, which is 7 in the present invention;

[0029] After the convolution operation at each scale, a residual connection is added to directly add the convolutional features to the original input features, keeping the channels of the original input features unchanged, fusing the convolutional features and the input features, and merging the two feature representations through the residual connection operation:

[0030] x=[Conv(x),identity(x)]

[0031] Among them, identity(x) means that the input feature x is directly passed without any operation, that is, the input feature is retained.

[0032] The fused multi-scale features are expanded back to the original number of channels through linear projection. The fully connected layer is also used to perform linear transformation on the fused features. The weight matrix and bias are used to weight and add the features to generate a feature representation with the same number of channels as the input features:

[0033] F=FC 2 ([x 1 ,x 2 ,x 3 ])

[0034] Among them, FC 2 represents the second fully connected layer, which maps the concatenated multi-scale features back to the target number of channels.

[0035] In step S3, the fused multi-scale features are input into the GPNR (Generalizable patch-based neural rendering) model for training. The function of the GPNR model is to receive multi-view fusion features as input and generate feature expressions of the target view using neural rendering technology. By processing the input features, GPNR can effectively reconstruct and complete the missing information in sparse scenes, and finally output high-quality new view images under sparse conditions, providing complete and consistent feature support for subsequent synthesis and presentation. Reference paper: Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer. pp. 156–174.

[0036] In step S5, the pre-trained model is used as the initialization of the new model, the weights and bias parameters of the pre-trained model are loaded, and used as the initial parameter settings of the new scene, and then fast fine-tuning is performed on the new scene data under sparse perspective. During the fine-tuning process, the parameters θ of the pre-trained model are used. 0 As the initial value, update the model by minimizing the loss function

[0037] Type parameters, Is a weighted loss function, expressed as:

[0038]

[0039] in represents the loss of the pre-trained model, Represents the loss term for new scene data. With a small number of camera perspectives, the model can quickly adapt to new scene features by updating the parameter θ through the gradient descent method, thereby accelerating the training and rendering process and improving the quality of new scene perspective synthesis.

[0040] Beneficial effects:

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention proposes a sparse new perspective synthesis method based on multi-scale feature fusion, which mainly combines multi-scale feature processing with residual mechanism to extract depth features. Compared with the prior art, this method makes full use of multi-scale features and effectively combines residual structure. Introducing multi-receptive field convolution and residual connection in multi-scale feature fusion enhances the feature expression ability, enables the model to capture depth information more accurately, and thus improves the rendering quality and global consistency in sparse perspective scenes. In addition, by selectively unfreezing the key layers of the pre-trained model for fine-tuning, the model can quickly adapt to sparse perspective scenes, significantly improving the training efficiency and adaptability. Description of the drawings:

[0043] Figure 1 This is a network framework diagram of the sparse new perspective synthesis method based on multi-scale feature fusion of the present invention.

[0044] Figure 2 is a flow chart of the method of the present invention.

[0045] Figure 3 The figures are the comparison results of the present invention, variant models and the most advanced methods in different scenarios. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with specific embodiments and accompanying drawings:

[0047] Example 1

[0048] Task Definition

[0049] Assume that a set of sparsely sampled multi-view images is input Each image represents a different camera perspective, and the COLMAP tool is used to accurately calculate the camera's posture, including the camera's internal and external reference matrices. The camera posture provides the position and orientation information of the camera in three-dimensional space, determines the perspective of the input image, and can generate images observed from different angles and directions by adjusting the camera posture. The goal of this invention is to achieve new perspective reconstruction under sparse input images through a neural rendering method, significantly improving the reconstruction quality.

[0050] The present invention is a sparse new perspective synthesis method based on multi-scale feature fusion, which has the following steps:

[0051] (1) Multi-scale reference point generation and feature sampling

[0052] First, according to the resolution H,W of the input image, a multi-scale spatial shape is generated for the subsequent segmentation and reshaping of multi-scale features.

[0053] The spatial shape of the multi-scale feature map is defined as:

[0054] (H i ,W i )=[(H×2,W×2),(H,W),(H / 2,W / 2)]

[0055] These spatial shapes (H i ,W i ) describes the size of each scale feature map, which are high resolution, medium resolution, and low resolution respectively.

[0056] For each feature map scale, reference points are generated by a uniform grid for spatial positioning of multi-scale features. i and i Defined as:

[0057] x i =[0.5,1.5,2.5,…,W i -0.5],y i =[0.5,1.5,2.5,…,H i -0.5]

[0058] Generate a two-dimensional grid and expand it into one dimension. The x- and y-coordinates of the reference point are:

[0059]

[0060] Normalize the coordinates of all reference points to obtain the normalized coordinates:

[0061]

[0062] Merge the reference points of all scales to form a unified reference point set. scales =3 corresponds to three scales, and the final reference point set is:

[0063]

[0064] Among them, i=1, 2, 3 correspond to three scales.

[0065] Next, the starting index of each scale is calculated to locate the position of each scale feature in the flattened feature. The calculation formula of the starting index is:

[0066] L 0 =0

[0067]

[0068] Then, using these reference points and the starting index, the flattened features are mapped back to the multi-scale feature map. Given the flattened feature matrix C∈R B×N×C , where B is the batch size, C is the number of channels.

[0069] For each scale i, extract the corresponding feature fragment:

[0070] start i =L i-1

[0071] end i =L i

[0072] C i =C[:,start i :end i ,:]

[0073] Next, multi-scale feature fusion is performed on the feature maps of each scale to generate a multi-scale feature pyramid:

[0074] Pyramid={C 0 ,C 1 ,C 2}

[0075] It is then input into the feature extraction network for subsequent multi-receptive field feature extraction and fusion.

[0076] (2) Multi-receptive field residual feature extraction

[0077] A multi-receptive field residual feature extraction module is applied on the multi-scale feature pyramid to further extract and fuse deep features. First, a fully connected layer is applied to perform a linear transformation on the input features:

[0078] X i =FC 1 (X i )

[0079] For each scale feature (X i )(i=1,2,3), firstly change X i The channel dimension is evenly divided into two parts:

[0080]

[0081] Here, X i1 and X i2 Represent the two sub-tensors after segmentation, and the shape of each sub-tensor is That is, it is equally divided along the channel dimension.

[0082] Then a depthwise separable convolution with different kernel sizes is applied to each sub-tensor:

[0083] Y i1 =Conv 3×3 (X i1 )

[0084] Y i2 =Conv 5×5 (X i2 )

[0085] Among them, Conv k×k represents a depthwise separable convolution with kernel size k×k.

[0086] Then, the convolutional features are concatenated in the channel dimension:

[0087] Y i =[Y i1 ,Y i2 ]

[0088] Note that the concatenation operation here is performed along the channel dimension.

[0089] Apply activation functions (such as GELU) and batch normalization operations. First, apply the GELU activation function to the concatenated features to enhance the nonlinear expression capability, and then perform batch normalization:

[0090] Z i =BN i sGELU(Y i )x

[0091] Among them, the activation function GELU is defined as:

[0092]

[0093] The mathematical representation of batch normalization is:

[0094]

[0095] Among them, μ i represents the average value of the i-th scale feature, σ i represents the variance of the i-th scale feature; γ i and β i is the training parameter, the initial value γ i =1,β i=0.

[0096] Then, the processed features are concatenated with the initial input features in the channel dimension (residual connection):

[0097] Z′ i =[Z i ,X i ]

[0098] Finally, the features of all scales are concatenated along the channel dimension to form the final feature representation:

[0099] Z=[Z′ 1 ,Z′ 2 ]

[0100] At this time, Z∈R B×H×W×2C , where B is the batch size, H and W are the height and width of the feature map, and C is the number of channels. The last layer is the second fully connected layer, which maps the features back to the output dimension:

[0101] Z out =FC 2 (Z)

[0102] The role of the second fully connected layer is to map the high-dimensional features of the network to the required output dimension for subsequent feature aggregation tasks.

[0103] (3) Transformer-based multi-view feature aggregation model

[0104] In new perspective synthesis, the GPNR model realizes new perspective image synthesis under sparse perspective conditions through Transformer-based multi-perspective feature aggregation. Its core idea is to use the multi-head self-attention mechanism to capture the global relationship between different perspectives, and combine geometric consistency modeling to gradually aggregate depth and feature information. Reference paper: Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer. pp. 156–174.

[0105] The present invention inputs the obtained deep features into the GPNR network to perform multi-view feature aggregation and new view image generation:

[0106] F output =GPNR(Z out )

[0107] Among them, F outputIt is a feature representation used to predict the target perspective and can be used to further generate images from new perspectives.

[0108] (4) Generate target image

[0109] The target feature representation output by GPNR is input into a multi-layer perceptron (MLP) to predict the color of the target ray:

[0110]

[0111] (5) Loss function design

[0112] In order to improve the performance of the model, GPNR designed a fine loss and regularization loss based on color supervision to enhance the ability to fit the color and texture of the target scene while suppressing the risk of overfitting:

[0113] Fine loss It is used to measure the color difference between the model predicted image and the real image, which is defined as:

[0114]

[0115] in, is the predicted pixel color value, p i is the true pixel color value, and N is the total number of pixels in the image, which is the same variable as N in the above text.

[0116] Regularization Loss By limiting the size of the model weights, the risk of overfitting is reduced, which is defined as:

[0117]

[0118] Wherein, λ is the regularization coefficient. In the present invention, λ is set to 0.01, and w represents the trainable weight of the model.

[0119] The final total loss function is:

[0120]

[0121] Wherein, α is a weight factor, which is set to 100 in the present invention to balance the relative importance of the fine loss and the regularization loss.

[0122] This loss design effectively improves the quality of the model's new-perspective image synthesis under sparse perspective conditions, while ensuring the training stability and generalization ability of the model.

[0123] (7) Pre-trained model fine-tuning

[0124] The pre-trained GPNR model is used as the basis. It is a general model obtained by pre-training on a large-scale dataset. Then, the present invention uses a small number of images with known viewpoints and their corresponding geometric information to perform fine-tuning in the target scene. During the fine-tuning process, the weights related to some high-level feature aggregation modules and geometric consistency modeling in the pre-trained model are frozen, and only the parameters of the underlying feature extraction module are trained to adapt to the color and texture characteristics of the target scene:

[0125]

[0126] in represents the loss of the pre-trained model, Represents the loss term for new scene data. With a small number of camera perspectives, the model can quickly adapt to new scene features by updating the parameter θ through the gradient descent method, thereby accelerating the training and rendering process and improving the quality of new scene perspective synthesis.

[0127] Example 2

[0128] like Figures 1-2 As shown, the present invention provides a sparse new perspective synthesis method based on multi-scale feature fusion, comprising the following steps:

[0129] Step 1: Pre-training of new view synthesis model

[0130] Prepare and preprocess diverse image datasets, train the model on a large-scale, general dataset, use the loss function to measure the error and update the weights through backpropagation, repeat the training until convergence, and combine iterative training of multiple datasets to improve generalization ability. Finally, save the optimized model weights to provide general features and initial parameters for sparse new perspective synthesis tasks.

[0131] Step 2: Multi-scale reference point generation and feature sampling

[0132] First, for the input sparse view image, multi-scale reference points are generated according to its resolution and feature sampling is performed. In this process, by calculating multi-scale spatial shapes, such as (H×2, W×2), (H, W), (H / 2, W / 2), a resolution basis is provided for feature maps of different scales. A two-dimensional reference point grid is generated using a grid generation method. After normalization, the reference point coordinates are mapped to the range of [0, 1] to form spatial information to guide sampling. Subsequently, based on these reference points, feature samples are performed on feature maps of different resolutions to extract multi-scale features of high, medium, and low resolutions. These features are fused into a unified feature representation through a splicing operation, providing a basis for subsequent processing.

[0133] Step 3: Multi-receptive field residual feature extraction

[0134] In the multi-scale feature fusion stage, the present invention designs a feature processing module based on multi-receptive field convolution and residual mechanism. The multi-receptive field convolution module uses deep convolution operations with different convolution kernels such as 3×3, 5×5 and k×k to extract local and global context information for high, medium and low resolution features. After each convolution operation, the information integrity of the input features is maintained by adding residual connections, and they are fused with the convolved features. The fused multi-scale features are further processed by activation functions and dimensionality reduction operations, and finally reshaped into a unified feature representation with richer expressive power, laying a solid foundation for the generation of new perspective images.

[0135] Step 4: Transformer model training based on multi-view feature aggregation

[0136] In the training stage based on multi-view feature aggregation, the present invention introduces the GPNR model, with Transformer as the core module, to gradually realize the efficient aggregation and fusion of multi-view information. The fused multi-scale features are first input into the self-attention mechanism to extract the global correlation between features from different perspectives. Subsequently, through geometric modeling, the multi-view features are further integrated in the epipolar dimension to capture the depth consistency information of the ray direction. In the stage of fusing the reference view features, the present invention adopts a weighted summation mechanism to generate the final target ray features according to the attention weight distribution for feature reconstruction of the target perspective.

[0137] Step 5: Fine-tune the pre-trained model:

[0138] In the pre-training model fine-tuning stage, the present invention uses an optimization method based on transfer learning to accelerate the training and rendering of new scenes. In the pre-training stage, the GPNR model is trained using large-scale multi-view image data to learn the fusion and reconstruction capabilities of multi-view features. For specific sparse view scenes, the model parameters are further updated through fine-tuning to make it more suitable for specific scene data. In the fine-tuning process, two loss functions are used: fine prediction loss and regularization loss Avoid model overfitting. In practical applications, fewer parameters are fine-tuned, so the training speed is greatly improved, and the quality of the generated results is significantly improved.

[0139] Step 6: Output the rendered new perspective image:

[0140] Finally, the target ray features generated by the above processing are input into the rendering module to generate a new perspective image. Specifically, the model reconstructs the features under sparse perspective conditions based on the color prediction of the target ray, forms the pixel values ​​under the target perspective, and combines all rays to generate a complete new perspective image. The present invention achieves high-quality new perspective image generation under sparse perspective conditions by combining multi-scale feature fusion with multi-perspective information modeling, significantly improves the generalization ability and generation efficiency of the model, and reduces the density requirement of perspective sampling, which has a wide range of practical application value.

[0141] Step 1 only needs to be performed once in this embodiment, while steps 2 to 4 are an iterative process. Finally, a new perspective image is obtained through step 5. The stopping condition of the iteration is to run on the NVIDIA GeForce RTX 2070 GPU for 20 hours or the overall loss function reaches 3×10 within 1000 iterations. -3 within the threshold.

[0142] Experimental Results

[0143] 1. Dataset

[0144] This experiment uses the Co3D dataset, which contains rich categories and objects and is therefore widely used for multi-view image generation and rendering tasks. In the experimental design, we selected a variety of scenes, each of which uses six images taken from different angles for training, rendering, and evaluation. Each rendering generates three images from different perspectives, and then these rendering results are compared with the real images. See the literature: Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, AC, 2017. Transformation-grounded image generation network for novel 3dview synthesis, in: Proceedings of the ieee conference on computer vision and pattern recognition, pp. 3500–3509.

[0145] To ensure the accuracy of image generation, we used the COLMAP tool to generate the camera pose of the image. The camera pose determines the angle from which the scene is observed and directly affects the viewing angle and quality of the rendered image. By adjusting the camera pose, we achieved the generation of images from multiple perspectives, thereby testing the generalization ability and rendering performance of the model in multi-perspective generation.

[0146] 2. Experimental Setup

[0147] The experiment was developed based on the JAX framework and trained using the Adam optimizer. The initial learning rate was set to η 0 =2×10 -5 The training time for each scene is approximately 20 hours to ensure the best performance under limited resource conditions. We run the experiments on an NVIDIA GeForce RTX 2070 GPU with a batch size of 8. Six images are selected for input with uniform viewpoints in each scene, and three images with new viewpoints are generated.

[0148] This experiment selected Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) as the main evaluation indicators to measure the quality of generated images. These indicators can comprehensively reflect the accuracy, structural similarity and visual effect between the generated images and the real images.

[0149] 3. Performance comparison

[0150] We compare our baseline model with the full model and recent state-of-the-art methods as follows:

[0151] GPNR[Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022a. Generalizable patch-based neural rendering, in: European Conference on Computer Vision, Springer. pp. 156–174.]: It is a neural radiation field model based on local patch features. It uses a patch-based neural network rendering method to decompose complex three-dimensional scenes into multiple small blocks for rendering.

[0152] LFNR[Suhail, M., Esteves, C., Sigal, L., Makadia, A., 2022b. Light field neural rendering, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 8269–8279.]: It is a neural rendering model based on light field representation for sparse perspectives. By introducing light field technology to directly model the relationship between perspective and light, it avoids the complex volume rendering calculations in traditional NeRF, thereby achieving more efficient rendering speed and better new perspective synthesis effects.

[0153] WAH[Bao, Y., Li, Y., Huo, J., Ding, T., Liang, X., Li, W., Gao, Y., 2023. Where and how: Mitigating confusion in neural radiance fields from sparse inputs, in: Proceedings of the 31st ACM International Conference on Multimedia, pp. 2180–2188.]: It is a window-aware hashing acceleration method for neural radiance fields, focusing on optimizing training efficiency and inference performance in sparse sampling scenarios. By introducing a window-aware hashing mechanism, WAH can effectively capture local features while avoiding the resource waste of traditional global hashing methods under sparse view conditions, thereby achieving efficient scene modeling and fast new view rendering.

[0154] (1) Comparison results of the method proposed in this paper with other methods in different scenarios

[0155] Table 1 Performance comparison of different methods in different scenarios

[0156]

[0157]

[0158] By comparing the characteristics and goals of all methods, in all scenarios, all indicators of the method of the present invention are superior to other comparison methods, and the rendered images are more realistic and clear. As shown in Table 1, the method of the present invention has achieved significant performance improvements in various scenarios, especially in the PSNR and SSIM indicators. Compared with other methods (such as LFNR, WAH, GPNR), the method of the present invention has an average improvement of 4-14 values ​​in PSNR value, SSIM has also reached a higher value, and the LPIPS value has been significantly reduced, indicating that it has obvious advantages in detail retention, structure restoration and visual perception quality of image rendering. These results demonstrate the excellent performance of the method of the present invention in image rendering tasks, and can achieve higher image quality and consistency in a variety of scenarios.

[0159] like Figure 3 As shown, in the three scenes of plants, books and bears, the method of the present invention is compared with GPNR, LFNR and WAH, and different images after rendering are obtained. In the plant scene, the method of the present invention can clearly show the texture and edge of the leaves, while other methods appear to be relatively blurred, especially LFNR and WAH, which can hardly see the details of the leaves and the real texture of the background. In the book scene, the method of the present invention accurately restores the text and page details of the book cover, while other methods have defects in detail level and color saturation. GPNR does not render well in details and colors, the text is not clear enough, LFNR is blurred and distorted, and WAH's rendering loses almost all details and is difficult to identify. In the bear scene, the method of the present invention shows the real texture of the hair and accurately restores the complete photo frame in the background, and the edges and internal contents of the photo frame can be clearly seen. These experimental results show that the method of the present invention has significant advantages in improving the quality of image rendering, especially in the case of sparse input number, it can better maintain global information and improve the quality of local details.

[0160] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A sparse new-view image synthesis method based on multi-scale feature fusion, characterized in that: The following steps are involved: S1. Multi-scale reference point generation and feature sampling: Generate reference points of different scales according to the resolution of the input image, and use these reference points for sampling on the corresponding multi-scale feature map, extract features at each scale and splice them to obtain initial feature information; S2. Multi-receptive field residual feature extraction: Using multi-receptive field convolution and residual connection mechanism, deep convolution operations with different convolution kernel sizes are performed on the sampled initial feature information, and information of different scales is obtained through multiple receptive fields. At the same time, residual connections are added to maintain the original features and fuse the features at each scale, and finally residual features of multiple receptive fields are generated; S3. Feature aggregation and image generation based on attention network: The fused multi-scale deep features are used as input and processed layer by layer through the self-attention module in the GPNR model. The multi-view information is gradually aggregated and aligned by capturing the global dependencies between different viewpoints, completing the deep fusion and expression of features, thereby generating new viewpoint images in sparse scenes. S4. Pre-training of new perspective synthesis model: The model is pre-trained on a large-scale, general-purpose dataset. By training a large number of dense images of different scenes and combining different objects in multiple datasets for multiple iterations of training, it covers various possible visual scenes and object types. At the same time, the model weights are optimized by loss function calculation and back propagation, so that the model can learn general visual features and high-level semantic features suitable for sparse new perspective synthesis tasks. S5. Pre-trained model fine-tuning: Further optimization is performed through transfer learning based on pre-trained models, the parameters of the pre-trained models are transferred to new scenes, and fine-tuned based on the new scene data under sparse perspectives. The model parameters are updated through back propagation, thereby accelerating the training and rendering of new scenes, making the model more suitable for specific sparse scenes. The resulting model can be used to generate images from new perspectives.

2. The sparse new perspective synthesis method based on multi-scale feature fusion according to claim 1, characterized in that: In step S1, firstly, reference points of multiple scales are generated according to the resolution of the input image. The reference points are key points used to identify the pixel positions in the feature map. The image is divided into a grid of fixed size by an automatic selection method. The center point of the grid is used as the reference point. After the coordinates are normalized, the reference point sets of different scales are merged. Then, feature sampling is performed on the corresponding multi-scale feature map (C1, C2, C3). The index position is located in the flattened feature matrix by the normalized coordinates of the reference point, and the eigenvalues ​​are extracted from it and mapped back to the multi-scale structure. These sampled multi-scale features are spliced ​​and fused into a unified feature representation C. This process is expressed as: C=[Sample(C1),Sample(C2),Sample(C3)] Where Sample(·) represents the multi-scale feature sampling based on the reference point, and C contains the fused multi-scale features, providing rich multi-scale information for subsequent processing.

3. The sparse new perspective synthesis method based on multi-scale feature fusion according to claim 1, characterized in that: In step S2, the specific steps of extracting the multi-receptive field residual features are as follows: First, the features C from different scales are linearly transformed through the fully connected layer, different weights and biases are applied to each channel, the input features are re-weighted and combined through matrix operations, the feature relationships between channels are adjusted and reorganized, and a linearly transformed feature representation is generated: FC1(C)=W·C+b Among them, C is the multi-scale feature generated in step S1, W and b are parameters obtained after training to help the model better fit the data, and FC1 represents the first fully connected layer; Use multi-scale receptive field convolution to perform deep convolution operations on the features after dimensionality reduction. Use convolution kernels of different sizes to convolve the features separately. Split the features in the channel dimension and convolve them one by one. Use convolution kernels of multiple receptive fields to perform independent deep convolution processing on the features of different parts. The specific formula is: Conv(x)=[Conv 3×3 (x1),Conv 5×5 (x2),Conv k×k (x3)] Among them, x1, x2, x3 are the three parts into which the input feature x is divided in the channel dimension. The channel dimension represents the number of features extracted by the network. Then, different convolution kernel sizes (3×3), (5×5), and (k×k) are used to perform convolution operations on (x1, x2, x3) to generate features of different scales. Conv represents convolution operation; After the convolution operation at each scale, a residual connection is added to directly add the convolutional features to the original input features, keeping the channels of the original input features unchanged, fusing the convolutional features and the input features, and merging the two feature representations through the residual connection operation: x=[Conv(x),identity(x)] Among them, identity(x) means that the input feature x is directly passed without any operation, that is, the input feature is retained. The fused multi-scale features are expanded back to the original number of channels through linear projection. The fully connected layer is also used to perform linear transformation on the fused features. The weight matrix and bias are used to weight and add the features to generate a feature representation with the same number of channels as the input features: F=FC2([x1,x2,x3]) Among them, FC2 represents the second fully connected layer, which maps the concatenated multi-scale features back to the target number of channels.

4. The sparse new perspective synthesis method according to claim 1, characterized in that: In step S3, the fused multi-scale features are input into the GPNR (Generalizable patch-based neural rendering) model for training. The function of the GPNR model is to receive multi-view fusion features as input and generate feature expressions of the target view using neural rendering technology. By processing the input features, GPNR can effectively reconstruct and complete the missing information in sparse scenes, and finally output high-quality new view images under sparse conditions, providing complete and consistent feature support for subsequent synthesis and presentation.

5. The sparse new perspective synthesis method according to claim 1, characterized in that: In step S5, the pre-trained model is used as the initialization of the new model, the weights and bias parameters of the pre-trained model are loaded, and used as the initial parameter settings of the new scene, and then fast fine-tuning is performed on the new scene data under sparse perspective. During the fine-tuning process, the parameters θ0 of the pre-trained model are used as the initial values, and the model parameters are updated by minimizing the loss function. Is a weighted loss function, expressed as: in represents the loss of the pre-trained model, Represents the loss term for new scene data. With a small number of camera perspectives, the model can quickly adapt to new scene features by updating the parameter θ through the gradient descent method, thereby accelerating the training and rendering process and improving the quality of new scene perspective synthesis.

Images