Panoramic image generation method based on cross-view feature interaction and neural radiation field

Abstract

The invention discloses a panoramic image generation method based on cross-view feature interaction and a neural radiation field. The panoramic image generation method comprises the following steps: performing multi-scale feature extraction on each acquired view source image; performing cross-view feature aggregation and interaction on the same-scale feature maps of all views to generate enhanced multi-scale enhanced features, and then performing feature fusion and decoding output to form alignment perception features; constructing 3D scene representation through a continuous plane neural radiation field on the basis of the alignment perception features and the multi-view-angle source image, then projecting a 3D scene to a coordinate system of a target view angle by adopting homography distortion, and obtaining a preliminary distortion image aligned with the target view angle through volume rendering; and finally, under the condition of aligning the perception characteristics, fusing and repairing the initial distorted images of all the visual angles into a high-quality seamless panoramic image. According to the method, the limitation of a traditional method under the scenes of large view angle difference, complex shielding and illumination variation is overcome, and high-quality fusion and reconstruction of multi-view-angle images are realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a panoramic image generation method based on cross-view feature interaction and neural radiation field. Background Technology

[0002] With the rapid development of artificial intelligence technology, seamless fusion of multi-view images and panoramic image generation technology, as a core research direction in computer vision and graphics, has significant application value in immersive content creation, digital twins, and autonomous driving environmental perception. In recent years, deep learning-based methods have significantly improved the realism of view synthesis through end-to-end differentiable rendering. Currently, researchers are enhancing the robustness of multi-view alignment by introducing attention mechanisms to strengthen cross-view feature associations or combining deformable convolutions to handle non-rigid deformations. Furthermore, the combination of implicit neural representations and volume rendering can model the continuous geometry and appearance of complex scenes, achieving high-quality new view generation. To address the challenges of geometric misalignment, occlusion holes, and inconsistent lighting in multi-view images, existing research has proposed a series of improvement methods, mainly including feature alignment optimization, geometric modeling paradigms, and image fusion strategies. For feature alignment optimization, researchers dynamically establish cross-view feature correspondences through deformable attention mechanisms, adaptively aggregating multi-view information. This method enables the network to achieve non-rigid deformation alignment in the feature space, thereby improving the matching effect in scenes with parallax. In addition, by constructing multi-layer planar scene representations or continuous neural radiation fields and combining them with differentiable volume rendering to achieve precise geometric deformation, dense and consistent 3D scenes can be reconstructed from multi-view images. This method significantly improves the geometric consistency of view synthesis, making the reasoning of occluded areas more reasonable. Furthermore, to address artifacts and discontinuities in image fusion, the introduction of generative adversarial networks and perceptual loss for joint optimization achieves pixel-level illumination unification and detail restoration, which is also a commonly used post-processing method.

[0003] While existing methods have made some progress in multi-view fusion and panoramic image generation, significant shortcomings remain. First, feature alignment methods based on traditional geometric transformations struggle to handle large parallax and complex occlusions, leading to ghosting and tearing at the fusion boundaries. Second, discrete 3D reconstruction methods such as multi-layer planar representations have limitations in geometric continuity, making it difficult to accurately model subtle structural changes. Furthermore, most methods treat feature alignment, geometric reconstruction, and image fusion as independent stages, lacking global optimization and resulting in error accumulation. Finally, existing fusion networks typically rely on supervised ground truth data, or struggle to guarantee a balance between geometric accuracy and visual realism in unsupervised settings.

[0004] Therefore, designing an end-to-end unsupervised framework that deeply integrates high-precision cross-view feature alignment, continuous scene geometry reconstruction, and perception-driven image generation is particularly important for achieving high-quality and robust panoramic image generation. Summary of the Invention

[0005] To address the aforementioned technical problems in existing technologies, this invention proposes an unsupervised panoramic image generation method based on cross-view feature interaction and neural radiation fields, the specific technical solution of which is as follows:

[0006] A panoramic image generation method based on cross-view feature interaction and neural radiation field includes:

[0007] Step 1: Obtain source images from different perspectives;

[0008] Step 2: For each source image from different viewpoints, extract image features using a multi-scale feature extraction network to obtain a multi-scale feature map corresponding to each viewpoint;

[0009] Step 3: For the same-scale feature maps from all perspectives, aggregate and interact through the cross-perspective feature interaction module, so that the feature maps of each perspective integrate the most relevant information from other perspectives to generate multi-scale enhanced features after cross-perspective attention enhancement. The multi-scale enhanced features are then output through feature fusion and decoding to form a feature representation with geometric alignment awareness, that is, the alignment-aware feature is obtained.

[0010] Step 4: Based on alignment-aware features and multiple viewpoint source images, a 3D scene representation consisting of multiple parallel depth planes is constructed through a continuous planar neural radiation field. Then, homography distortion is used to project the 3D scene onto the coordinate system of the target viewpoint, and a preliminary distorted image aligned with the target viewpoint is obtained through volume rendering.

[0011] Step 5: An unsupervised panoramic image generation module consisting of a low-resolution deformation branch and a high-resolution thinning branch is used to fuse and repair the preliminary distorted images from each viewpoint into a seamless panoramic image by using alignment-aware features as conditions.

[0012] Furthermore, in step 2, the multi-scale feature extraction network uses a ResNet-50 with the global average pooling layer and fully connected layer removed as the encoder to extract features in four stages. The output of the last residual block of each stage is taken as the feature map to obtain feature maps of different scales in the four stages. A feature pyramid network is used as the decoder to generate a set of multi-scale feature maps for each viewpoint.

[0013] Furthermore, the ResNet-50 first preprocesses the input image, performing initial downsampling and feature extraction through a 7x7 convolutional layer and a 3x3 max pooling layer. Then, it sequentially processes the image through four stages of ResNet-50: the first stage contains 3 residual blocks, maintaining the original size of the output feature map but increasing the number of channels to 256; the second stage contains 4 residual blocks, halving the spatial size by setting the stride of the first residual block to 2, and increasing the number of channels to 512; the third stage contains 6 residual blocks, again downsampling through the first residual block, further halving the output feature map size, and increasing the number of channels to 1024; the fourth stage contains 3 residual blocks, downsampling again to obtain the deepest feature map; in each stage, the output of the last residual block is extracted as the feature representation for that stage, thus obtaining four feature maps at different scales.

[0014] Furthermore, the feature pyramid network acquires feature maps of different scales from the four stages of ResNet-50 output. Through top-down paths and lateral connections, it fuses the strong semantic features of the high-level layers with the fine positional features of the low-level layers. Starting from the deepest layer, it upsamples the features of the previous layer by a factor of 2. The upsampled features are added to the ResNet features of the same layer that have undergone 1x1 convolution for dimensionality reduction. Then, a 3x3 convolution is used to eliminate the aliasing effect of upsampling, finally obtaining a feature pyramid with a unified number of 256 channels. For each viewpoint, a set of multi-scale features is obtained.

[0015] Furthermore, step 3 specifically includes:

[0016] Step 3.1: For the feature map of viewpoint r at scale l For one of the feature points p, a query vector q is generated for that point using learnable linear projection weights. The query vector q is then used by an offset network to learn and generate an adaptive spatial offset. ;

[0017] Step 3.2: For all viewpoints s in the feature map at scale l, take the initial position of feature point p as the center, sample K neighboring feature points around it, and generate the corresponding key vector k and value vector v for each sampled point through linear projection.

[0018] Step 3.3: Calculate the attention weights for the query vector q of feature point p in view r and the key vectors of all K sampling points in all view s one by one, and perform Softmax normalization on the dimensions of all view and all sampling points to ensure that the sum of the weights is 1. Based on the attention weights, perform a weighted summation of the sampling values ​​of all view to obtain the new feature of feature point p after information enhancement. By performing the above operation on each point on the feature map, cross-view information interaction is completed point by point, and finally, multi-scale enhanced features after cross-view attention enhancement are generated for each view.

[0019] Step 3.4: Input the multi-scale enhanced features of each view after cross-view attention enhancement. Through the reverse process of the feature pyramid network, starting from the deepest layer, gradually upsample and fuse with higher resolution layers to obtain high-resolution fused features. Then, use a lightweight decoder to predict the final output alignment-aware features and alignment confidence map.

[0020] Furthermore, in step 3.4, the multi-scale enhancement features of the input feature pyramid network start from the deepest layer of the network. First, a 1x1 convolution is used to adjust the number of channels and serve as the starting point for fusion. Then, upsampling and fusion are performed level by level. For each layer, the low-resolution feature map fused from the previous layer is upsampled to match the spatial size of the feature map of the current layer. The upsampled features are then fused with the enhancement features of the current layer after dimensionality reduction by a 1x1 convolution. After fusion, a non-linear activation function and a 3x3 convolution are used to smooth the fusion result. This process is repeated, and the fused feature map obtained at the highest resolution layer is the high-resolution fused feature. Finally, a lightweight decoder, using the high-resolution fused feature as input, predicts the final output alignment-aware features and alignment confidence map.

[0021] Furthermore, step 4 specifically includes:

[0022] Step 4.1: Input the source images from each viewpoint and their corresponding alignment-aware features;

[0023] Step 4.2: Transform the image pixel coordinates to real 3D coordinates, and randomly sample several depth planes within the view frustum. Then, use a hierarchical sampling strategy to perform high-frequency encoding on each disparity in the disparity space to obtain depth conditions and enhance the network's learning of high-frequency details.

[0024] Step 4.3: Based on the input of Step 4.1, the source image features are extracted by the encoder and fused with the alignment-aware features, and then sent to the decoder. The decoder takes the encoded depth conditions and fused features as input and outputs a 4-channel depth plane containing color and volume density information.

[0025] Step 4.4: Based on the color and volume density information, use volume rendering to synthesize the source viewpoint image and depth map;

[0026] Step 4.5: Using homography distortion, for each pixel in the target panoramic image, the sampling points corresponding to each depth plane in the source image are calculated using the relative pose between the source view and the target view. After sampling the color and density information under the target view, the distorted image is synthesized through volume rendering.

[0027] Furthermore, in step 4, the reconstruction of the continuous planar neural radiation field is supervised by rendering loss and edge-aware depth smoothing loss.

[0028] Furthermore, step 5 specifically includes:

[0029] Step 5.1: Perform confidence-based weighted fusion of the preliminary distorted images and corresponding alignment-aware features from each viewpoint to obtain the initial fused image and initial fused features. Then, downsample to a low resolution to obtain the downsampled fused image and downsampled fused features.

[0030] Step 5.2: Input the downsampled fused image and downsampled fused features into the low-resolution deformation branch. First, it is downsampled through a multi-layer encoder with each layer containing convolution, normalization and ReLU activation functions. Based on the convolution and stride, the feature map size is halved and the number of channels is doubled. Then, it is upsampled through a SPADE conditional decoder. The output of the last layer of the decoder is scaled to the range [−1,1] by a Tanh activation function to obtain a low-resolution result sketch. The low-resolution result sketch is a preliminary panoramic image with correct structure but blurred details.

[0031] Step 5.3: After upsampling the low-resolution result sketch to the original high resolution, the original high-resolution preliminary fused image is used to stitch it together with the image in the channel dimension. Then, together with the initial fused features, the image is processed through multiple SPADE residual blocks to achieve detail enhancement. The output of the last SPADE residual block is processed through a convolutional layer and a Tanh activation function to obtain a high-resolution detail map.

[0032] Step 5.4: Input the high-resolution detail image into a lightweight, refined network consisting of convolutional layers for pixel-level fine-tuning, and finally output a seamless panoramic image of the target viewpoint.

[0033] Furthermore, the core of the SPADE conditional decoder upsampling path is the SPADE residual block. The internal operation of the SPADE residual block involves normalizing the input features using SPADE and performing convolution and activation function operations, ultimately resulting in residual connections.

[0034] Beneficial effects: The method of this invention deeply integrates multi-view geometric alignment cues and neural scene representation to produce geometrically consistent and visually realistic panoramic results. It overcomes the limitations of traditional methods in scenarios with large differences in viewpoints, complex occlusion, and changing lighting. Furthermore, it adopts an unsupervised volume rendering and adversarial loss joint optimization strategy during training to achieve high-quality fusion and reconstruction of multi-view images. It is especially suitable for real-world scenarios with complex parallax, occlusion, and changing lighting. Attached Figure Description

[0035] Figure 1 This is a flowchart of an unsupervised panoramic image generation method based on cross-view feature interaction and neural radiation field according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the cross-view feature interaction module in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the continuous planar neural radiation field processing in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of the unsupervised panoramic image generation module in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0040] like Figure 1 As shown, this invention provides an unsupervised panoramic image generation method based on cross-view feature interaction and neural radiation field. It constructs and trains a three-stage progressive panoramic generation network, which mainly includes a cross-view feature interaction module, a continuous planar neural radiation field, and an unsupervised panoramic image generation module. The network is optimized using a cross-view attention mechanism and a geometric perception loss function. Finally, the optimal model obtained through training achieves high-quality seamless panoramic image generation. The specific steps include:

[0041] Step 1: Collect source images from different perspectives as input.

[0042] Step 2: Process source images from each viewpoint using a multi-scale feature extraction network. This network uses a ResNet-50 with the global average pooling layer and fully connected layers removed as the encoder, extracts features in four stages, takes the output of the last residual block of each stage as the feature map, and uses a Feature Pyramid Network (FPN) as the decoder to generate a set of multi-scale feature maps for each viewpoint.

[0043] The ResNet-50 first preprocesses the input image, performing initial downsampling and feature extraction through a 7x7 convolutional layer and a 3x3 max-pooling layer to obtain the first-stage feature map. Subsequently, the image features are processed sequentially through four core stages of ResNet-50: the first stage contains 3 residual blocks, maintaining the original output feature map size but increasing the number of channels to 256; the second stage contains 4 residual blocks, halving the spatial size by setting the stride of the first residual block to 2, and increasing the number of channels to 512; the third stage contains 6 residual blocks, again downsampling through the first residual block, further halving the output feature map size, and increasing the number of channels to 1024; the fourth stage contains 3 residual blocks, downsampling again to obtain the deepest feature map. In each stage, the output of the last residual block is extracted as the feature representation for that stage, thus obtaining feature maps at four different scales. .

[0044] The feature pyramid network obtains the four-stage feature map output by ResNet-50. The feature pyramid network fuses strong semantic features from high levels with fine positional features from low levels through a top-down path and lateral connections. Starting from the deepest layer, it upsamples the features of the previous layer by a factor of 2. The upsampled features are then added to the ResNet features in the same layer that have undergone 1x1 convolutional dimensionality reduction. A 3x3 convolution is then used to eliminate the aliasing effect of the upsampling, ultimately resulting in a feature pyramid with a uniform number of 256 channels. For each viewpoint i, a set of multi-scale features is obtained. .

[0045] Step 3: Information is fused through the cross-view feature interaction module. This module aggregates and interacts with features of the same scale from all viewpoints, enabling each viewpoint to integrate the most relevant information from other viewpoints and generate enhanced cross-view features. These cross-view features are then further fused and decoded to form a feature representation with geometric alignment awareness, i.e., alignment-aware features.

[0046] Among them, such as Figure 2 As shown, the implementation of the cross-view feature interaction module includes the following steps:

[0047] Step 3.1: For a given reference viewpoint r, the feature map at a specific scale l A feature point p is selected, and a query vector q for that point is generated using learnable linear projection weights. The query vector q is then input into an offset network to learn and generate an adaptive spatial offset. .

[0048]

[0049]

[0050] in, This is the feature map of viewpoint r at the l-th scale, where the superscript indicates the viewpoint number and the subscript indicates the scale. In the feature map The eigenvector at coordinate p. It is a learnable linear projection weight matrix whose function is to project the original feature vector onto a new space to generate a query. q is the generated query vector, which encodes the semantic information of point p and is used for matching with information from other perspectives. It is a lightweight network architecture. It is a two-dimensional offset, representing the offset of point p relative to the initial position on the feature map of view s, where s is the index of the source view and k is the index of the sampling point. For each source view, we sample K points.

[0051] Step 3.2: For the feature map of all viewpoints s (including the reference viewpoint r itself) on the same scale, sample K neighboring feature points around the initial position of feature point p. For each sampled point, generate the corresponding key vector k and value vector v through linear projection.

[0052]

[0053]

[0054] in, It is on the feature map of all viewpoints s The coordinates of the kth sampling point It is achieved through bilinear interpolation. The feature vector sampled at that coordinate. , It is a learnable linear projective weight matrix. It is the key vector of the k-th sampling point from the s-th viewpoint, representing the "tag" or "keyword" of the content at that location, used to calculate the relevance with query q. It is the value vector of the kth sampling point from the s-th viewpoint. It contains the actual information content of that location and is the feature to be aggregated in the end.

[0055] Step 3.3: The query vector q of feature point p in view r is compared with the key vectors of all K sampling points in all source views s to calculate attention weights, and softmax normalization is performed on all dimensions of all views and all sampling points to ensure that the sum of the weights is 1. Finally, the sample value vectors of all source views are weighted and fused based on this weight to obtain the new feature of feature point p after information enhancement. By repeating the above operation at each position in the feature map, cross-view information interaction can be completed point by point, and finally a set of multi-scale enhanced features after cross-view attention enhancement is generated for each view i. .

[0056]

[0057]

[0058] in, It calculates the query vector q and the key vector. The dot product of these two factors yields their correlation score; a larger dot product indicates a higher correlation. It is a scaling factor used to prevent the dot product result from becoming too large, which would cause the gradient of the Softmax function to become smaller. It's a normalization operation that performs a Softmax calculation on all N viewpoints and all K sampling points (a total of N×K scores) to ensure that the sum of all weights is 1. It is the final attention weight, which is a scalar. It is a new feature enhanced from point p. By performing the above operation on each point on the feature map, a multi-scale enhanced feature after cross-view attention enhancement is obtained for each view i.

[0059] The multi-scale enhanced features after cross-view attention enhancement are further fused and decoded to form a feature representation with geometric alignment awareness, i.e., alignment-aware features. Specifically, this is achieved through the reverse process of the feature pyramid network, starting from the deepest layer and gradually upsampling and fusing with higher resolution layers. The multi-scale enhanced features from each viewpoint after cross-view attention enhancement are then input. Starting from the deepest layer 5, a 1x1 convolution is first used to adjust the number of channels, serving as the starting point for fusion. Then, upsampling and fusion are performed layer by layer. This is a repetitive process. For each layer 1 (from 4 to 2), the lower-resolution feature map fused from the previous layer is upsampled to match the spatial size of the current layer's feature map. The upsampled features are then fused with the enhanced features from the current layer after dimensionality reduction via a 1x1 convolution. After fusion, a non-linear activation function and a 3x3 convolution are applied to smooth the fusion result, eliminating aliasing effects that may be caused by upsampling and enhancing non-linear expressive power. This process is repeated until the fused feature map is obtained at the highest resolution layer (layer 2). The required high-resolution fusion features Finally, using a lightweight decoder and high-resolution fused features as input, the alignment-aware features of the final output are predicted. Alignment Confidence Plot This provides powerful geometric and semantic priors for subsequent stages.

[0060] Step 4: Based on alignment-aware features and multi-view source images, a dense 3D scene representation is constructed through a continuous planar neural radiation field, achieving accurate and occlusion-handling geometric mapping. The 3D scene representation consists of a series of planes parallel to the image plane at different depths. Each plane stores its RGB color and volume density information, thereby establishing an accurate and occlusion-handling geometric model and rendering a roughly aligned panoramic view, providing richer and more geometrically consistent information for subsequent stages.

[0061] Among them, such as Figure 3 As shown, the realization of a continuous planar neural radiation field includes the following steps:

[0062] Step 4.1: Input source images from each viewpoint and its corresponding alignment-aware features ;

[0063] Step 4.2: Convert the image pixel coordinates to true 3D coordinates to provide a basis for subsequent 3D distance calculations, within the specified view frustum range. N depth planes are randomly sampled within the space. A hierarchical sampling strategy is employed in the disparity space. For each parallax Perform high-frequency encoding to generate depth conditions This enhances the network's ability to learn high-frequency details.

[0064]

[0065]

[0066]

[0067] Where x and y are the pixel coordinates of the point on the image plane, and s is the depth value of the point relative to the camera's optical center. It is the inverse of the intrinsic parameter matrix, and T represents the coordinate transformation operation. , These are the nearest and farthest disparity boundaries, respectively. N is the total number of sampled planes, and U represents the uniform distribution within the interval [a, b]. It is the disparity value of the plane index i to be sampled. , It is a series of frequencies, where sin and cos are sine and cosine functions, and L is the number of frequency levels in the code. It is a hyperparameter positional encoding function that maps to a high-dimensional vector;

[0068] Step 4.3: Extract source image features using the encoder, fuse these features with alignment-aware features, and then feed them into the decoder. The decoder takes the encoded depth conditions and fused features as input and outputs a 4-channel depth plane containing color and volume density information.

[0069]

[0070] in, It is the input source view image. It is an encoder network architecture. It is the output source view image feature. It is the alignment-aware feature of the input. It is a feature fusion operation. It is the depth condition of the output of the previous stage of the input. It is a decoder network architecture. It is the RGB color value of the corresponding pixel on the i-th plane. It is the volume density of the corresponding pixel on the i-th plane;

[0071] Step 4.4: Based on color and volume density information, synthesize the source view image using volume rendering formulas. and depth map As a self-supervised truth value for subsequent unsupervised learning.

[0072]

[0073] in, It is the 3D spatial distance between the i-th plane and the (i+1)-th plane, which is calculated by the coordinate transformation function T. It is the opacity of the i-th plane. Cumulative transmittance represents the probability that a ray successfully propagates from the first plane to the i-th plane without being blocked. It is the source view image obtained through rendering and synthesis. It is a depth map obtained by rendering and synthesizing, used to implement unsupervised learning;

[0074] Step 4.5: To obtain an image distorted to the target's viewpoint, homography warping is used to project the reconstructed 3D scene onto the target coordinate system. For pixels in the target panoramic image... Utilizing the relative pose between the source view and the target view Calculate the sampling points corresponding to each depth plane in the source image. After sampling the color and density fields from the target viewpoint, the final distorted image is synthesized using a volume rendering formula. .

[0075]

[0076] in, , These are the pixel coordinates on the target's viewpoint image. , R and t are the camera intrinsic parameter matrices for the source and target viewpoints, respectively, and R and t are the rotation matrix and translation vector from the target viewpoint coordinate system to the source viewpoint coordinate system. This is the depth value of the current plane. , These are the calculated pixel coordinates on the i-th depth plane of the source viewpoint. , It samples the color and density in the target coordinate system. Based on the distorted field, the volume rendering formula of the previous stage is used. , Perform volume rendering to obtain the final distorted image.

[0077] The entire process of the continuous planar neural radiation field is supervised by rendering loss and edge-aware depth smoothing loss, with the goal of making the rendered image and depth map as realistic and accurate as possible.

[0078] Step 5: Using an unsupervised panoramic image generation module consisting of low-resolution and high-resolution branches, and taking alignment perception features as a condition, multiple preliminary distorted images output by the continuous planar neural radiation field that have been roughly aligned with the target viewpoint are fused and repaired, and finally a seamless, high-definition and highly realistic panoramic image is synthesized.

[0079] Among them, such as Figure 4 As shown, the implementation of the unsupervised panoramic image generation module includes the following steps:

[0080] Step 5.1: Initially distort the images from each viewpoint With corresponding alignment-aware features Confidence-based weighted fusion is performed to obtain the initial fused image. With fusion features The image is then downsampled to a lower resolution to generate a downsampled fused image. Features fused with downsampling .

[0081]

[0082]

[0083]

[0084]

[0085] Where N is the total number of source views, and i is the view index. These are pixel coordinates on the image. It is the initial distorted image of the i-th viewpoint at pixels RGB color values ​​at that location It is the alignment confidence map of the i-th viewpoint at the pixel level. scalar value at that point It is a very small positive number, called the smoothing term, the purpose of which is to prevent the denominator from being zero. This is the initial reference image after fusion. It is the feature vector of the distortion alignment perception feature of the i-th viewpoint at position (x,y). It is the fused conditional feature map. It is a downsampling operation. It is a low-resolution image obtained after downsampling. It is a low-resolution conditional feature map obtained after downsampling;

[0086] Step 5.2: Subsample and fuse the images With features The input is a low-resolution deformation branch, whose goal is to learn the global content layout quickly and stably. The low-resolution branch employs an encoder-decoder architecture. The input first passes through a multi-layer encoder downsampling path, with each layer containing convolutions, normalization, and ReLU activation functions. Based on the convolutions and stride, the feature map size is halved and the number of channels is doubled. Next, it passes through a SPADE-conditional decoder upsampling path. The output of the last layer of the decoder is scaled to the [−1,1] range by a Tanh activation function, resulting in a low-resolution sketch. , It is a preliminary panoramic view with a correct structure but lacking in detail. The image has a reasonable overall structure but lacks detail.

[0087]

[0088]

[0089]

[0090]

[0091] in, It is the input image. It is the output feature map of the previous layer. It is a two-dimensional convolutional layer. It is an instance normalization layer. It is a modified linear unit activation function. It is the layer above the decoder (layer l-1). The output feature map is a feature map with a spatially enlarged size. This refers to the feature map taken from the corresponding layer of the encoder path. The l-th layer of the decoder corresponds to the (L-l)-th layer of the encoder, where L is the total number of layers in the encoder. It involves skip connections and splicing operations to fuse multi-scale information. The low-resolution conditional feature map obtained after downsampling. It is a SPADE residual block operation. This is the output of the final layer residual block. It is an activation function. It is a preliminary panoramic view with a correct structure but blurry details, which serves as a low-resolution sketch for the final output.

[0092] In step 5.2, the core of the SPADE conditional decoder upsampling path is the SPADE residual block. The internal operation of the SPADE residual block involves normalizing the input features using SPADE, performing convolution and activation function operations, and finally performing residual connections, as detailed below:

[0093]

[0094]

[0095]

[0096] in, , Calculate input features Mean and standard deviation for each channel and each sample. , It is a modulation parameter generation function, derived from conditional features. Tensors obtained through dynamic prediction by convolutional layers These are the output features of the first processing. It is a two-dimensional convolutional layer. It is a modified linear unit activation function. These are intermediate features after processing. It is a shortcut path processing feature. This is a characteristic of the final output of the SPADE residual block operation;

[0097] Step 5.3: Result sketch at low resolution Guided by the structure, the branches are refined at high resolution, and meticulously crafted on the original resolution to restore lifelike details.

[0098] low-resolution sketch Upsampling to high resolution and the initial fused image with the original resolution. The data is stitched together along the channel dimension and then combined with the initial fusion features. The data is fed into multiple SPADE residual blocks for detail enhancement. Finally, after convolution and Tanh activation, a high-resolution detail map is output. .

[0099]

[0100]

[0101]

[0102]

[0103] in, It is a low-resolution sketch of the input result. It is an upsampling operation. It is the original high-resolution preliminary fused image. It is a channel-level splicing operation. The output is the initial feature used for processing the input multi-layer SPADE residual blocks. These are initial fusion features. This is the SPADE residual block operation, whose internal operations are exactly the same as the SPADE residual block operation in the low-resolution branch. This is the output after residual block processing. This is the output of the last SPADE residual block. It is a two-dimensional convolutional layer. It is an activation function. It is the final output high-resolution detail image.

[0104] In step 5.3, the internal operations of the SPADE residual block of the high-resolution refined branch are exactly the same as those of the SPADE residual block of the low-resolution branch, but the input and condition features are high-resolution.

[0105] Step 5.4: Upgrade the high-resolution detail image Input a lightweight convolutional refinement network for pixel-level fine-tuning to eliminate possible artifacts and discontinuities, ultimately generating a high-quality, seamless panoramic image of the target viewpoint.

[0106]

[0107] in, It is a lightweight, refined network consisting of convolutional layers, which can focus on pixel-level fine-tuning. It is the final output, which is the desired panoramic image of the target view.

[0108] The low-resolution deformation branch loss includes content loss and seam loss, focusing on learning the correct global structure and layout. The high-resolution refinement branch loss includes content loss, adversarial loss, and tolerance consistency loss, focusing on restoring fine details and textures. The loss function is designed to simultaneously ensure content fidelity and visual realism.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A panoramic image generation method based on cross-view feature interaction and neural radiance field, characterized in that, The method comprises the following steps: Step 1: obtaining source images of different perspectives; Step 2: extracting image features of each perspective source image through a multi-scale feature extraction network to obtain multi-scale feature maps corresponding to each perspective; Step 3: aggregating and interacting the same scale feature maps from all perspectives through a cross-perspective feature interaction module to integrate the most relevant information in other perspectives into the feature map of each perspective, and generating multi-scale enhanced features after cross-perspective attention enhancement, and then outputting the multi-scale enhanced features through feature fusion and decoding to form a feature representation with geometric alignment perception, i.e. obtaining an alignment perception feature; Step 4: based on the alignment perception feature and the multiple perspective source images, constructing a 3D scene representation composed of multiple parallel depth planes through a continuous planar neural radiance field, then projecting the 3D scene to the coordinate system of the target perspective through homography distortion, and obtaining a preliminary warped image aligned with the target perspective through volume rendering; Step 5: using an unsupervised panoramic image generation module composed of a low-resolution deformation branch and a high-resolution refinement branch, and using the alignment perception feature as a condition to fuse and repair the preliminary warped images of each perspective into a seamless panoramic image.

2. The panoramic image generation method according to claim 1, wherein In step 2, the multi-scale feature extraction network uses ResNet-50 without a global average pooling layer and a fully connected layer as an encoder, extracts features in four stages, takes the output of the last residual block in each stage as a feature map to obtain four feature maps of different scales, and uses a feature pyramid network as a decoder to generate a set of multi-scale feature maps for each perspective.

3. The panoramic image generation method according to claim 2, wherein The ResNet-50 first preprocesses the input image, performs preliminary downsampling and feature extraction through a 7x7 convolution layer and a 3x3 max pooling layer, and then sequentially passes through the four stages of ResNet-50: the first stage contains 3 residual blocks, the output feature map maintains the original size but the channel number increases to 256; the second stage contains 4 residual blocks, the spatial size is halved through the first residual block with a step of 2, and the channel number increases to 512; the third stage contains 6 residual blocks, also downsamples through the first residual block, the output feature map size is further halved, and the channel number increases to 1024; the fourth stage contains 3 residual blocks, and the deepest feature map is obtained after downsampling again; the output of the last residual block in each stage is extracted as the feature representation of that stage, thereby obtaining four feature maps of different scales.

4. The panoramic image generation method according to claim 2, wherein The feature pyramid network obtains the four-stage feature maps of different scales output by ResNet-50, fuses the strong semantic features of the high layer and the fine position features of the bottom layer through a top-down path and a horizontal connection, performs 2 times upsampling on the features of the upper layer from the deepest layer, adds the upsampled features and the ResNet features reduced in dimension through a 1x1 convolution of the same layer, and then eliminates the aliasing effect of upsampling through a 3x3 convolution, finally obtaining a feature pyramid with a unified channel number of 256; for each perspective, a set of multi-scale features is obtained.

5. The panoramic image generation method of claim 1, wherein, Step 3 specifically comprises: Step 3.1: For a feature map of view r on scale l The query vector q of one of the feature points p is generated by a learnable linear projection weight, and the adaptive spatial offset is learned and generated by the offset network from the query vector q ; Step 3.2: For all feature maps of scale l, take the initial position of the feature point p as the center, sample K neighboring feature points around it, and generate the corresponding key vectors k and value vectors v for each sampled point by linear projection respectively; Step 3.3: Calculate the attention weight between the query vector q of the feature point p in view r and the key vectors of all K sampled points in all views s, and perform Softmax normalization in the dimension of all views and all sampled points to ensure that the sum of the weights is 1. According to the attention weight, the sampling values of all views are weighted and summed to obtain the new features of the feature point p after information enhancement. By performing this step on each point on the feature map, cross-view information interaction is completed point by point, and finally multi-scale enhanced features after cross-view attention enhancement are generated for each view; Step 3.4: Input the multi-scale enhanced features of each view after cross-view attention enhancement, and through the reverse process of the feature pyramid network, start from the deepest layer, gradually upsample and fuse with higher resolution layers to obtain high-resolution fusion features, and then predict the final output alignment perception features and alignment confidence map through the lightweight decoder.

6. The panoramic image generation method according to claim 5, wherein In step 3.4, the multi-scale enhanced features input into the feature pyramid network start from the deepest layer of the network. First, a 1x1 convolution is used to adjust the channel number, which serves as the starting point for fusion. Then, the features are gradually upsampled and fused level by level. For each layer, the low-resolution feature map from the previous level of fusion is upsampled to match the spatial size of the current layer's feature map. The upsampled feature is then fused with the enhanced feature of the current layer after 1x1 convolution dimension reduction. After fusion, a non-linear activation function and a 3x3 convolution are followed to smooth the fusion results. This process is repeated to obtain the high-resolution fusion feature at the highest resolution layer. Finally, the high-resolution fusion feature is input into the lightweight decoder to predict the final output alignment perception features and alignment confidence map.

7. The panoramic image generation method of claim 1, wherein, The step 4 specifically includes: Step 4.1: Input the source images of each view and their corresponding alignment perception features; Step 4.2: Convert the image pixel coordinates to real 3D coordinates, randomly sample several depth planes within the view frustum, and then use a hierarchical sampling strategy to encode each disparity in disparity space at high frequency to obtain depth conditions to enhance the network's learning of high-frequency details; Step 4.3: Based on the input of step 4.1, extract the source image features through the encoder, then perform feature fusion with the alignment perception features, and finally input them into the decoder. The decoder takes the encoded depth conditions and fused features as input and outputs a 4-channel depth plane containing color and volume density information; Step 4.4: Use volume rendering to synthesize the source view images and depth maps based on the color and volume density information; Step 4.5: Use homography warping. For pixels in the target panoramic image, calculate their corresponding sampling points on the source image depth planes using the relative pose between the source view and the target view. After sampling the color and density information in the target view, synthesize the warped image through volume rendering.

8. The panoramic image generation method of claim 1, wherein, In step 4, the reconstruction of the continuous planar neural radiance field is supervised by the rendering loss and the edge-aware depth smoothness loss.

9. The panoramic image generation method of claim 1, wherein, The step 5 specifically comprises: Step 5.1: confidence-based weighted fusion of the preliminary warped images of each view and the corresponding alignment-aware features to obtain an initial fusion image and an initial fusion feature, and then down-sampling to a low resolution to obtain a down-sampled fusion image and a down-sampled fusion feature; Step 5.2: inputting the down-sampled fusion image and the down-sampled fusion feature into the low-resolution deformation branch, first passing through a multi-layer encoder down-sampling path containing convolution, normalization and ReLU activation function in each layer, then passing through a SPADE-conditioned decoder up-sampling path, and outputting the last layer of the decoder through a Tanh activation function to scale to the range of [-1, 1] to obtain a low-resolution result sketch, which is a preliminary panoramic image with correct structure but fuzzy details; Step 5.3: after up-sampling the low-resolution result sketch to the original high resolution, the original high-resolution preliminary fusion image is spliced in the channel dimension, and then processed through a multi-layer SPADE residual block together with the initial fusion feature to realize detail enhancement, and the output of the last SPADE residual block is obtained through a convolution layer and a Tanh activation function to obtain a high-resolution detail map; Step 5.4: inputting the high-resolution detail map into a lightweight refining network composed of convolution layers for pixel-level fine-tuning to finally output a seamless panoramic image of the target view.

10. The panoramic image generation method according to claim 9, wherein The core of the SPADE-conditioned decoder up-sampling path is the SPADE residual block, and the internal operation of the SPADE residual block is to normalize the input features using SPADE, and then perform convolution and activation function operations, and finally perform residual connection.

Images