Few-sample new-view-angle image synthesis method based on multi-scale mixed perception and state space cooperation

Abstract

The invention relates to a few-sample new-view-angle image synthesis method based on multi-scale mixed perception and state space collaboration, and solves the problems of depth ambiguity, excessive smooth texture and serious artifacts in a shielding region caused by insufficient geometric constraints in a generalized neural radiation field in a few-sample scene compared with the prior art. And the problems of high quadratic calculation complexity, large video memory occupation and low rendering efficiency caused by introducing 3D Transform to perform global modeling in the prior art are solved. And the defect of inaccurate geometric surface positioning caused by noise interference in the traditional sampling strategy based on attention weight is overcome. The method comprises the following steps of data set construction and sparse input acquisition; setting a hierarchical bidirectional feature aggregation network; multi-scale features are extracted; carrying out global context modeling; extracting and fusing local geometric features; and generating a new view angle image. According to the method, multi-scale features are extracted through the hierarchical bidirectional feature aggregation network, collaborative modeling of global context and local geometry is combined, depth ambiguity caused by insufficient geometric constraints under the condition of few samples is effectively solved, and the method is especially suitable for being used under the limited observation condition that only sparse source view angle images are provided. And performing high-quality new view rendering and image generation on an unknown scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision and graphics technology, specifically to a method for synthesizing novel perspective images with few samples based on multi-scale hybrid perception and state-space collaboration. Background Technology

[0002] With the rapid development of fields such as virtual reality, intelligent driving, cultural heritage protection, game development, architectural design, and film and television production, high-quality 3D scene reconstruction and novel perspective generation technologies have become research hotspots. However, traditional 3D reconstruction methods face numerous challenges in handling complex lighting, occlusion, and textureless regions, and typically require a large number of high-quality input views, which is often difficult to achieve in practical applications. For example, in some complex environments, such as disaster sites, inaccessible areas, or protected cultural heritage sites, obtaining dense views is very difficult or prohibitively expensive; in medical imaging, such as X-ray or CT scans, the number of imaging sessions needs to be minimized to reduce radiation exposure to patients; in security checks, such as airport security checks, rapid and efficient scanning is required without disrupting normal procedures; in autonomous vehicles, it may not be possible to obtain complete perspective data in all situations, thus requiring the inference of complete scene information from a limited number of perspectives. Neural Radiation Field (NeRF), as a deep learning-based scene representation method, can generate high-quality novel perspective images from sparse input views, but it is highly dependent on the number and distribution of input views. To overcome the limitations of traditional neural radiation fields, which require individual optimization for each scene and rely on dense input viewpoints, generalizable neural radiation fields have emerged. These methods utilize multi-view geometric constraints (such as epipolar geometry) and pre-trained networks to synthesize new viewpoint images of unknown scenes without retraining, and perform well when there is sufficient multi-view input.

[0003] However, in scenarios with very few input viewpoints (e.g., only 1 to 3 images), existing generalizable neural radiation fields still face some serious problems. Current mainstream methods mainly rely on local geometric features extracted along epipolar lines to infer depth and color. When viewpoints are sparse, this local dependency is easily susceptible to depth ambiguity due to occlusion, reflections, or texture repetition. Furthermore, due to a lack of awareness of the global scene context, the model cannot effectively utilize non-local information between images for completion, leading to artifacts, geometric collapse, or overly smoothed textures in the synthesized image.

[0004] Therefore, how to enable generalizable neural radiation fields to synthesize new perspective images of unknown scenes with the highest possible quality even in scenarios with few samples has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the problems of depth ambiguity, excessive texture smoothing, and severe artifacts in occluded areas caused by insufficient geometric constraints in generalizable neural radiation fields in few-sample scenarios in existing technologies; the high quadratic computational complexity, large memory consumption, and low rendering efficiency caused by introducing 3D Transformers for global modeling in existing technologies; and the defects of inaccurate geometric surface localization caused by noise interference in traditional attention weight-based sampling strategies. This invention provides a novel few-sample perspective image synthesis method based on multi-scale hybrid perception and state space collaboration to solve the above problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination includes the following steps:

[0008] 11) Dataset construction and sparse input acquisition: Obtain a sparse source view image set of the scene to be reconstructed and the camera pose parameters corresponding to each image;

[0009] 12) Define a hierarchical bidirectional feature aggregation network: Define a hierarchical bidirectional feature aggregation network including an encoder for feature extraction, a multi-scale self-attention module for cross-scale feature enhancement, and a decoder for fine features;

[0010] 13) Extraction of multi-scale features;

[0011] 14) Global context modeling;

[0012] 15) Extraction and fusion of local geometric features;

[0013] 16) Generation of new perspective images: The final ray features output from the ray Transformer are input into the multilayer perceptron decoder to synthesize a new perspective image; the final target ray features are then processed... Input a multilayer perceptron decoder The RGB color value corresponding to the target ray is obtained by mapping. :

[0014] ,

[0015] The calculated RGB color values ​​of each target ray Arrange the pixel coordinates of the new perspective to be synthesized to generate a complete new perspective image.

[0016] In the dataset construction and sparse input acquisition steps, for the target new viewpoint to be synthesized, its spatial distance with all source viewpoints is calculated, and the closest viewpoint is selected from the sparse source viewpoint image set. The image is used as the source view image, and the source view image and the new target view image are combined to form a source-target view pair to establish the relative geometric projection relationship between the source view and the target view.

[0017] Includes the following steps:

[0018] 21) Acquire multiple source view images of the scene to be reconstructed, and pre-calculate the camera intrinsic and extrinsic parameters of each image using the structure-in-motion algorithm. The intrinsic parameters include focal length and principal point, and the extrinsic parameters include rotation matrix and position vector. For static backgrounds and dynamic objects in the images, size normalization is uniformly performed.

[0019] 22) For each new viewpoint to be synthesized, calculate its spatial Euclidean distance to all source viewpoints:

[0020] Based on pre-calculated camera extrinsics, obtain the position vector of the target from a new perspective. With the Position vector from the source perspective The position vector represents the three-dimensional coordinates of the camera's optical center in the world coordinate system.

[0021] Spatial Euclidean distance between the two The calculation formula is:

[0022] ,

[0023] in, , , Represent the target's new perspective position vector, respectively. In the world coordinate system axis, shaft and Spatial coordinate components of the axis, , , They represent the first Individual source perspective position vector In the world coordinate system axis, shaft and Spatial coordinate components of the axis, It is a three-dimensional real vector space;

[0024] Based on the calculated spatial Euclidean distance Sort all source views in ascending order and select the top ones. Each corresponding image constructs a source-target view pair.

[0025] The extraction of multi-scale features involves: using a hierarchical bidirectional feature aggregation network to extract features from the source viewpoint image, generating a multi-scale feature map for each source viewpoint image. The multi-scale feature map is divided into a first-resolution feature map and a second-resolution feature map; this includes the following steps:

[0026] 31) Input the source view image into the encoder.

[0027] The encoder consists of stacked convolutional neural network modules forming the basic feature extraction unit. The processing flow of the convolutional neural network module can be formally represented as follows:

[0028] ,

[0029] Where Input represents the input, Conv represents the convolutional layer, BN represents the batch normalization layer, and ReLU is the activation function. Represents output characteristics;

[0030] Based on the convolutional neural network module, a cascaded architecture is used to extract multi-scale features step by step. The specific process is as follows:

[0031] The first stage generates C1: the source view image, with a size of [size missing]. , Image height, The image width is first downsampled, and then features are extracted using three stacked convolutional neural network modules to generate a resolution equal to the original. Figure 4 One-third of the first-stage feature map C1, with a size of ;

[0032] The second stage generates C2: The first-stage feature map C1 is used as input for further downsampling, and then further abstracted features are extracted using four stacked convolutional neural network modules, generating a second-stage feature map C2 with a resolution one-eighth that of the original image, and a size of [size missing]. ;

[0033] The third stage generates C3: The second-stage feature map C2 is used as input, downsampled again, and deep feature extraction is performed through six stacked convolutional neural network modules to generate a third-stage feature map C3 with a resolution of one-sixteenth of the original image. The size is... ;

[0034] The feature maps {C1, C2, C3} generated in the above three stages together constitute the multi-scale coding feature pyramid of the encoder;

[0035] 32) Cross-scale feature enhancement:

[0036] A multi-scale self-attention module is used to perform cross-scale interaction on the encoded feature maps at different stages of the encoder. The multi-scale self-attention module downsamples the three-stage feature maps generated by the encoder to a uniform size and generates a query matrix through linear transformation. Key matrix Sum matrix ,

[0037] To explore the potential positional relationships and long-distance dependencies between objects at different scales, a self-attention mechanism is applied to calculate the weights, as shown in the following formula:

[0038] ,

[0039] ,

[0040] ,

[0041] ,

[0042] in, and Feature maps representing different scales. This is the weight matrix. For scaling parameters, The generated self-attention feature map;

[0043] 33) Define the core processing components in the decoder:

[0044] The decoder contains an information integration attention module and a global and local feature co-fusion module, which work together to restore feature resolution and generate multi-scale feature maps.

[0045] 34) Cascaded decoding and output of multi-scale feature maps:

[0046] 341) Construct an information integration attention module;

[0047] 342) Constructing a local feature collaborative fusion module

[0048] 343) Generate the second resolution feature map: The feature map output by the multi-scale self-attention module is processed by the global and local feature fusion module and then upsampled. It is then concatenated with the second-stage feature map C2 and processed by the information integration attention module and the global and local feature fusion module in sequence. Finally, the second resolution feature map is output through 1×1 convolution.

[0049] 344) Generate the first resolution feature map: After upsampling the second resolution feature map, it is concatenated with the first stage feature map C1, and then processed by the information integration attention module and the global and local feature fusion module in sequence. Finally, the first resolution feature map is output through 1×1 convolution.

[0050] The global context modeling involves: using the multi-axis sparse state space Mamba module to perform sequence modeling on the second-resolution feature map, constructing a global context feature vector containing cross-view dependencies; including the following steps:

[0051] 41) Define the adaptive size division algorithm:

[0052] For the input second-resolution feature map, its size is , For feature map height, Set the preset ideal partition size for the feature map width. , This indicates the preset division height. Indicates the preset partition width, and checks the feature map height. Is it based on the current preset division height? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ;

[0053] Check feature map width Is it subject to the current preset partition width? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ;

[0054] The final result This refers to an adaptive partitioning size without fill;

[0055] like or If reducing the size to 1 still fails to satisfy the integer division condition, then the corresponding block size or grid size is directly set to 1, at which point it degenerates into pixel-by-pixel processing;

[0056] 42) Set up local window state space scanning unit processing:

[0057] Calculate the adaptive block size using an adaptive size partitioning algorithm , This indicates the preset block height. This indicates the preset block width, and the feature map is reshaped and transposed. Shape tensors, Indicates the batch size of the input data. For feature map height, The width of the feature map. This represents the number of channel dimensions in the feature map, maintaining independence between blocks, and assigning a size of [missing information]. The local blocks are flattened into a one-dimensional sequence, and the Mamba state space model is independently applied for scanning to capture the pixel dependencies within the local region.

[0058] 43) Set global sparse grid state space scan cell processing:

[0059] Calculate the adaptive mesh size using an adaptive size partitioning algorithm , This indicates the preset grid height. This indicates the preset grid width, which reshapes the feature map into... Shape tensors, Indicates the batch size of the input data. This indicates the number of channel dimensions in the feature map. For feature map height, The feature map width is used as the basis for dimensional permutation, which aggregates pixels belonging to the same grid position.

[0060] Extract grid point sequences with extremely large spans and lengths of . Input state-space model;

[0061] 44) Configure cross-view collaborative state space scan unit processing:

[0062] Will The feature maps from each source viewpoint are stitched together along the viewpoint dimension to form a shape of... The tensor will Considering the sequence length, for each pixel at a spatial location, a state space model is applied to scan along the view axis to capture the geometric consistency and semantic association between different viewpoints;

[0063] 45) Global Feature Generation:

[0064] After the second-resolution feature map is processed through steps 42), 43), and 44), an implicit scene representation is obtained. Using the Transformer decoder, the target ray is... Position and orientation codes are used as queries in The search is performed to generate a global context feature vector for the target ray. :

[0065] ,

[0066] in, Indicates the decoder Transformer,

[0067] This global context feature vector contains macroscopic semantic information about the scene, providing prior guidance for subsequent geometric matching.

[0068] The extraction and fusion of local geometric features are as follows: using the global context feature vector as the query vector, the view Transformer aggregates features on the epipolar lines of the source view image and its corresponding first resolution feature map, and then the ray Transformer fuses the aggregated features on the epipolar lines with the position and direction information of the target ray along the direction of the target ray to obtain ray features; including the following steps:

[0069] 51) Epipolar sampling and feature projection: Based on the direction of the target ray, in Sampling is performed along the epipolar line on the image plane of each source viewpoint, and features are obtained from the first resolution feature map;

[0070] 52) Globally Guided Aggregation: To distinguish between occluded regions and the real surface, the global context feature vector generated in step 13) is used. As the query vector, the features of the epipolar sampling points obtained from sampling will be used. and its position code As the key and value, input to the Transformer view, where the position is encoded. The contribution weights of different source viewpoints to the target ray are calculated based on the relative geometric positions of the source viewpoint and the target ray, and then dynamically calculated using a self-attention mechanism. The formula is as follows:

[0071] ,

[0072] in Represents a view, such as a Transformer.

[0073] 53) Joint Feature Embedding: In order to integrate the macroscopic semantics and microscopic geometric details of the scene, the global context feature vector is... With local geometric features The data is concatenated along the channel dimension to form a joint feature embedding. :

[0074] ,in Indicates a splicing operation;

[0075] 54) Obtaining ray features: After embedding the joint features through a ray Transformer, the target ray representation is obtained. :

[0076] ;

[0077] in, Represents a light ray Transformer;

[0078] 55) Concentrate sampling points in the region that contributes the most to the final color;

[0079] 551) First, sample the light uniformly. For each sampling point, input the ray Transformer to obtain the original attention weights. This weight reflects the contribution of that depth point to the final color;

[0080] 552) Sigmoid Nonlinear Smoothing: To address the problem of unstable and noisy dynamic range of the original attention weights, the Sigmoid activation function is used to compress and smooth them, mapping them to the (0,1) interval.

[0081] 553) Probability density function generation: The smoothed discrete weights are fitted to a continuous probability density function along the depth of the light ray using the Gaussian kernel regression method. The calculation formula is:

[0082] ,

[0083] in, Let Gaussian kernel function be defined as:

[0084] ,

[0085] Indicates the target depth position along the target ray. Indicates the first Depth values ​​of uniformly sampled points Represents the natural constant. Pi This is the bandwidth parameter, used to control the smoothness of the probability density function;

[0086] 554) Fine sampling: Using the inverse transform sampling algorithm to additionally sample data from the probability density function. These are fine sample points that are automatically clustered on the surface of the object, i.e., the areas with high weight.

[0087] 555) Final Feature Aggregation: The coarse and fine sampling points are merged and input into the ray Transformer again for multi-layer self-attention aggregation, outputting the final target ray features. .

[0088] The construction of the information integration attention module includes the following steps:

[0089] 61) Two feature maps of the same resolution from the encoder and decoder are concatenated along the channel dimension to form a combined feature. ;

[0090] 62) respectively for At height and width Average pooling and max pooling are performed in the direction to generate channels. eigenvectors on , and ;

[0091] Through concat operation and two-dimensional convolution... One-dimensional convolution Batch normalized normalization (BN) and sigmoid activation function processing are used to obtain spatial attention weights in the height and width dimensions. and :

[0092] ,

[0093] ;

[0094] 63) Combine the generated weights with the combined features Multiplying and concatenating the results yields the final output of the information integration attention module:

[0095] ,

[0096] Among the symbols Indicates element-wise multiplication, symbol This indicates element-wise addition.

[0097] The construction of the global and local feature collaborative fusion module includes the following steps:

[0098] 71) Setting up local information branches:

[0099] Perform the following steps on the input feature maps respectively: Convolution and Convolution is performed, and the results of the two processes are summed to form initial local features; subsequently, cascaded dilated convolution processing is performed: a set of convolution dilation rates is defined. The feature maps are processed iteratively multiple times using depthwise separable convolutions, with each iteration employing a different dilation rate to abstract and represent objects of different sizes in various directions. Finally, all intermediate feature maps generated during the concatenation process are concatenated along the channel dimension and processed through a single... Convolution adjusts the number of channels to obtain the output of the local information branch. ;

[0100] 72) Define a global information branch: Use the state-space model as the main body to extract global features and model long-range contextual information.

[0101] First, the input feature map is uniformly divided into four sub-feature blocks along the channel dimension, denoted as... Secondly, each sub-feature block is input into the visual state space module for parallel processing, utilizing the selective scanning mechanism of the visual state space module to capture global dependencies. Finally, the four processed sub-feature blocks are reassembled along the channel dimension to obtain the output of the global information branch. ;

[0102] 73) Setting the feature embedding head:

[0103] To effectively embed local information into global information, a feature similarity-based fusion mechanism is designed, specifically as follows:

[0104] First, output the local branch. With global branch output By performing element-by-element addition, preliminary fusion characteristics are obtained. ;

[0105] Secondly, calculation and Cosine similarity between The formula is as follows:

[0106] ,

[0107] in, and These represent the pixel values ​​at the same location in the two feature maps. Total number of pixels For pixel index;

[0108] Next, using the Sigmoid function to... Perform nonlinear mapping to generate weight masks; perform preliminary feature fusion. conduct Convolution yields intermediate features ;Will The weighted features are obtained by multiplying them element-wise with the weight mask. ;

[0109] Finally, With the process After convolution By adding elements one by one, the residual characteristics are obtained. Interpolation and upsampling are performed to output the final enhanced feature map. Its computational logic is expressed as follows:

[0110]

[0111] ,

[0112] ,

[0113] ,

[0114] Among them, symbols Indicates element-wise multiplication, symbol This indicates element-wise addition. express convolution, This indicates an interpolation upsampling operation.

[0115] The epipolar sampling and feature projection include the following steps:

[0116] 81) Ray definition and spatial point sampling: For any target ray to be rendered It originates from the optical center. and direction vector Define the sampling depth range on the light source. ,in The near-end boundary value representing the sampling depth. Represents the far-end boundary value of the sampling depth, within which uniform or stratified sampling is performed. Discrete depth points At this time, the first Three-dimensional spatial sampling points The coordinates are represented as:

[0117] ;

[0118] 82) Coordinate projection transformation: using the pre-acquired first... Camera intrinsic parameter matrix from a single source perspective and extrinsic parameter matrix ,in Let be a rotation matrix. As a position vector, a point in three-dimensional space Projected to the On a two-dimensional image plane from a single source perspective;

[0119] Let the coordinates of the projected two-dimensional pixels be... The homogeneous coordinate transformation formula is:

[0120] ,

[0121] in As a scale factor, and These represent the x and y coordinates of the projection point on the source view image plane, respectively. Through this formula, each three-dimensional point on the target ray is mapped to a two-dimensional projection point on the source view image.

[0122] 83) Characteristic bilinear interpolation: Due to the calculated projected coordinates Typically, the value is a floating-point number, and bilinear interpolation is used to calculate the value from the first... First resolution feature map of the feature map from the source perspective The feature vector corresponding to the sampling point, for each sampling point The features extracted The calculation is as follows:

[0123] ;

[0124] in This represents the bilinear interpolation sampling algorithm.

[0125] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination.

[0126] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, enables a method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space collaboration.

[0127] Beneficial effects

[0128] The present invention provides a method for synthesizing novel perspective images with few samples based on multi-scale hybrid perception and state space collaboration. Compared with existing technologies, this method extracts multi-scale features through a hierarchical bidirectional feature aggregation network and combines global context and local geometry collaborative modeling. This effectively solves the depth ambiguity caused by insufficient geometric constraints under few sample conditions. It is especially suitable for high-quality novel perspective rendering and image generation of unknown scenes under limited observation conditions where only sparse source perspective images (such as 1 to 3 images) are provided.

[0129] Experiments show that, under extreme conditions with only 1-3 input viewpoints, the present invention outperforms the state-of-the-art methods in peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual similarity (LPIPS) on various datasets, and can clearly reconstruct high-frequency textures, reflective regions, and object edges.

[0130] This invention innovatively introduces the multi-axis sparse state space Mamba module to replace the traditional 3D Transformer for global context modeling. Because Mamba has linear computational complexity... Compared to the quadratic complexity of Transformer This invention significantly reduces memory usage and computational overhead when processing high-resolution feature maps, thereby achieving faster inference speed and higher rendering frame rate.

[0131] This invention replaces the traditional feature pyramid with a hierarchical bidirectional feature aggregation network. Through a multi-scale self-attention module, a global and local feature fusion module, and an information integration attention module, it overcomes the limited receptive field of traditional convolutional networks. This enables the model to capture long-range semantic dependencies even from a sparse perspective, improving the accuracy of feature matching, particularly in weakly textured regions.

[0132] The adaptive sampling strategy based on Sigmoid smoothing and kernel regression proposed in this invention effectively solves the problems of noisy, unstable dynamic range, and lack of physical meaning in the original attention weights. By transforming the weights into a smooth and physically consistent probability density function, this invention can accurately concentrate a limited number of sampling points within the depth range of the object's surface, thereby significantly improving the precision of the object's geometric reconstruction and reducing cloud-like artifacts during rendering.

[0133] The method proposed in this invention can quickly reconstruct high-quality 3D scene perspectives with extremely low input costs (only 1-3 photos are needed), and can be widely used in fields such as virtual reality content generation, augmented reality interaction, 3D display of e-commerce products, robot environmental perception, and sparse view inspection by drones. Attached Figure Description

[0134] Figure 1 This is a flowchart of a novel perspective synthesis method based on multi-scale hybrid sensing and state space collaboration according to the present invention.

[0135] Figure 2 This is a diagram of the hierarchical bidirectional feature aggregation network structure of the present invention.

[0136] Figure 3 The structure diagram of the multi-axis sparse state space Mamba module and the decoder Transformer module;

[0137] Figure 4 This is a schematic diagram comparing the synthesized new perspective images with real images in different scenarios according to an embodiment of the present invention. Detailed Implementation

[0138] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0139] like Figure 1 As shown, the method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state space collaboration according to the present invention includes the following steps:

[0140] The first step, dataset construction and sparse input acquisition, involves obtaining a sparse set of source view images of the scene to be reconstructed, along with the camera pose parameters for each image. For the target new viewpoint to be synthesized, its spatial distance to all source viewpoints is calculated, and the closest viewpoint is selected from the sparse source viewpoint image set. The image is used as the source view image, and the source view image and the new target view image are combined to form a source-target view pair to establish the relative geometric projection relationship between the source view and the target view.

[0141] (1) Obtain multiple source view images of the scene to be reconstructed, and use the motion reconstruction structure algorithm to pre-calculate the camera intrinsic and extrinsic parameters of each image. The intrinsic parameters include focal length and principal point, and the extrinsic parameters include rotation matrix and position vector. For static background and dynamic objects in the image, size normalization is uniformly performed.

[0142] (2) For each new viewpoint to be synthesized, calculate its spatial Euclidean distance with all source viewpoints:

[0143] Based on pre-calculated camera extrinsics, obtain the position vector of the target from a new perspective. With the Position vector from the source perspective The position vector represents the three-dimensional coordinates of the camera's optical center in the world coordinate system.

[0144] Spatial Euclidean distance between the two The calculation formula is:

[0145] ,

[0146] in, , , Represent the target's new perspective position vector, respectively. In the world coordinate system axis, shaft and Spatial coordinate components of the axis, , , They represent the first Individual source perspective position vector In the world coordinate system axis, shaft and Spatial coordinate components of the axis, It is a three-dimensional real vector space;

[0147] Based on the calculated spatial Euclidean distance Sort all source views in ascending order and select the top ones. The strategy of constructing source-target view pairs using only three source images ensures the physical proximity of the input image and the target viewpoint.

[0148] The second step is to set up a hierarchical bidirectional feature aggregation network: the hierarchical bidirectional feature aggregation network includes an encoder for feature extraction, a multi-scale self-attention module for cross-scale feature enhancement, and a decoder for fine features.

[0149] The hierarchical bidirectional feature aggregation network adopts a U-shaped architecture. The encoder consists of cascaded convolutional neural network modules, while the decoder gradually restores resolution through upsampling. Crucially, the multi-scale features from the encoder are passed to the decoder via skip connections. Along these skip connection paths, a global-local feature co-fusion module and an information integration attention module are embedded. The global-local feature co-fusion module enhances the global context before feature fusion, while the information integration attention module reconstructs spatial location information after feature fusion, thus achieving bidirectional feature aggregation and enhancement.

[0150] The third step is multi-scale feature extraction. A hierarchical bidirectional feature aggregation network is used to extract features from the source viewpoint images, generating multi-scale feature maps for each source viewpoint image. These multi-scale feature maps are divided into a first-resolution feature map and a second-resolution feature map. The structure of the hierarchical bidirectional feature aggregation network is as follows: Figure 2 As shown, the hierarchical bidirectional feature aggregation network aims to replace the traditional feature pyramid network and solve the problem of limited receptive field in convolutional neural networks.

[0151] (1) Input the source view image into the encoder.

[0152] The encoder consists of stacked convolutional neural network modules forming the basic feature extraction unit. The processing flow of the convolutional neural network module can be formally represented as follows:

[0153] ,

[0154] Where Input represents the input, Conv represents the convolutional layer, BN represents the batch normalization layer, and ReLU is the activation function. Represents output characteristics;

[0155] Based on the convolutional neural network module, a cascaded architecture is used to extract multi-scale features step by step. The specific process is as follows:

[0156] The first stage generates C1: the source view image, with a size of [size missing]. , Image height, The image width is first downsampled, and then features are extracted using three stacked convolutional neural network modules to generate a resolution equal to the original. Figure 4 One-third of the first-stage feature map C1, with a size of ;

[0157] The second stage generates C2: The first-stage feature map C1 is used as input for further downsampling, and then further abstracted features are extracted using four stacked convolutional neural network modules, generating a second-stage feature map C2 with a resolution one-eighth that of the original image, and a size of [size missing]. ;

[0158] The third stage generates C3: The second-stage feature map C2 is used as input, downsampled again, and deep feature extraction is performed through six stacked convolutional neural network modules to generate a third-stage feature map C3 with a resolution of one-sixteenth of the original image. The size is... ;

[0159] The feature maps {C1, C2, C3} generated in the above three stages together constitute the multi-scale coding feature pyramid of the encoder, providing a rich data foundation for subsequent cross-scale attention interactions.

[0160] (2) Cross-scale feature enhancement:

[0161] A multi-scale self-attention module is used to perform cross-scale interaction on the encoded feature maps at different stages of the encoder. The multi-scale self-attention module downsamples the three-stage feature maps generated by the encoder to a uniform size and generates a query matrix through linear transformation. Key matrix Sum matrix ,

[0162] To explore the potential positional relationships and long-distance dependencies between objects at different scales, a self-attention mechanism is applied to calculate the weights, as shown in the following formula:

[0163] ,

[0164] ,

[0165] ,

[0166] ,

[0167] in, and Feature maps representing different scales. This is the weight matrix. For scaling parameters, This is the generated self-attention feature map.

[0168] (3) Define the core processing components in the decoder:

[0169] The decoder contains an information integration attention module and a global and local feature co-fusion module, which work together to restore feature resolution and generate multi-scale feature maps.

[0170] (4) Cascaded decoding and output of multi-scale feature maps:

[0171] A1) Construct an information integration and attention module;

[0172] Two feature maps of the same resolution from the encoder and decoder are concatenated along the channel dimension to form a combined feature. ;

[0173] To each At height and width Average pooling and max pooling are performed in the direction to generate channels. eigenvectors on , and ;

[0174] Through concat operation and two-dimensional convolution... One-dimensional convolution Batch normalized normalization (BN) and sigmoid activation function processing are used to obtain spatial attention weights in the height and width dimensions. and :

[0175] ,

[0176] ;

[0177] The generated weights and combined features Multiplying and concatenating the results yields the final output of the information integration attention module:

[0178] ,

[0179] Among the symbols Indicates element-wise multiplication, symbol This indicates element-wise addition.

[0180] A2) Constructing a local feature collaborative fusion module:

[0181] Setting up local information branches:

[0182] Perform the following steps on the input feature maps respectively: Convolution and Convolution is performed, and the results of the two processes are summed to form initial local features; subsequently, cascaded dilated convolution processing is performed: a set of convolution dilation rates is defined. The feature maps are processed iteratively multiple times using depthwise separable convolutions, with each iteration employing a different dilation rate to abstract and represent objects of different sizes in various directions. Finally, all intermediate feature maps generated during the concatenation process are concatenated along the channel dimension and processed through a single... Convolution adjusts the number of channels to obtain the output of the local information branch. ;

[0183] Define a global information branch: Use the state-space model as the main body to extract global features and model long-range contextual information.

[0184] First, the input feature map is uniformly divided into four sub-feature blocks along the channel dimension, denoted as... Secondly, each sub-feature block is input into the visual state space module for parallel processing, utilizing the selective scanning mechanism of the visual state space module to capture global dependencies. Finally, the four processed sub-feature blocks are reassembled along the channel dimension to obtain the output of the global information branch. ;

[0185] Define the feature embedding head:

[0186] To effectively embed local information into global information, a feature similarity-based fusion mechanism is designed, specifically as follows:

[0187] First, output the local branch. With global branch output By performing element-by-element addition, preliminary fusion characteristics are obtained. ;

[0188] Secondly, calculation and Cosine similarity between The formula is as follows:

[0189] ,

[0190] in, and These represent the pixel values ​​at the same location in the two feature maps. Total number of pixels For pixel index;

[0191] Next, using the Sigmoid function to... Perform nonlinear mapping to generate weight masks; perform preliminary feature fusion. conduct Convolution yields intermediate features ;Will The weighted features are obtained by multiplying them element-wise with the weight mask. ;

[0192] Finally, With the process After convolution By adding elements one by one, the residual characteristics are obtained. Interpolation and upsampling are performed to output the final enhanced feature map. Its computational logic is expressed as follows:

[0193]

[0194] ,

[0195] ,

[0196] ,

[0197] Among them, symbols Indicates element-wise multiplication, symbol This indicates element-wise addition. express convolution, This indicates an interpolation upsampling operation.

[0198] A3) Generate the second resolution feature map: The feature map output by the multi-scale self-attention module is upsampled after being processed by the global and local feature fusion module, and then concatenated with the second-stage feature map C2. It is then processed by the information integration attention module and the global and local feature fusion module in sequence, and finally output as the second resolution feature map through 1×1 convolution.

[0199] A4) Generate the first resolution feature map: Upsample the second resolution feature map and concatenate it with the first stage feature map C1. Then, process it through the information integration attention module and the global and local feature fusion module in sequence. Finally, output the first resolution feature map through 1×1 convolution.

[0200] The fourth step is global context modeling. The multi-axis sparse state space Mamba module is used to perform sequence modeling on the second-resolution feature map to obtain an implicit scene representation. Then, the decoder Transformer uses the target ray as the query to retrieve and generate a global context feature vector containing cross-view dependencies from this implicit scene representation. The structures of the multi-axis sparse state space Mamba module and the decoder Transformer module are as follows: Figure 3 As shown, the traditional Transformer has high computational complexity. The problem is that this step introduces a computationally complex method with linear computational time complexity. The state-space model (Mamba).

[0201] (1) Set the adaptive size division algorithm:

[0202] For the input second-resolution feature map, its size is , For feature map height, Set the preset ideal partition size for the feature map width. , This indicates the preset division height. Indicates the preset partition width, and checks the feature map height. Is it based on the current preset division height? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ;

[0203] Check feature map width Is it subject to the current preset partition width? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ;

[0204] The final result This refers to an adaptive partitioning size without fill;

[0205] like or If reducing the size to 1 still fails to meet the divisibility condition, the corresponding block size or grid size is directly set to 1, at which point it degenerates into pixel-by-pixel processing.

[0206] (2) Set up local window state space scanning unit processing:

[0207] Calculate the adaptive block size using an adaptive size partitioning algorithm , This indicates the preset block height. This indicates the preset block width, and the feature map is reshaped and transposed. Shape tensors, Indicates the batch size of the input data. For feature map height, The width of the feature map. This represents the number of channel dimensions in the feature map, maintaining independence between blocks, and assigning a size of [missing information]. The local blocks are flattened into a one-dimensional sequence, and the Mamba state-space model is applied independently for scanning to capture the pixel dependencies within the local region.

[0208] (3) Set global sparse grid state space scan unit processing:

[0209] Calculate the adaptive mesh size using an adaptive size partitioning algorithm , This indicates the preset grid height. This indicates the preset grid width, which reshapes the feature map into... Shape tensors, Indicates the batch size of the input data. This indicates the number of channel dimensions in the feature map. For feature map height, The feature map width is used as the basis for dimensional permutation, which aggregates pixels belonging to the same grid position.

[0210] Extract grid point sequences with extremely large spans and lengths of . Input state-space model.

[0211] (4) Set up cross-view collaborative state space scanning unit processing:

[0212] Will The feature maps from each source viewpoint are stitched together along the viewpoint dimension to form a shape of... The tensor will Treating it as the sequence length, for each pixel at a spatial location, a state space model is applied to scan along the viewpoint axis to capture the geometric consistency and semantic association between different viewpoints.

[0213] (5) Global feature generation:

[0214] After the second-resolution feature map is processed through steps 42), 43), and 44), an implicit scene representation is obtained. Using the Transformer decoder, the target ray is... Position and orientation codes are used as queries in The search is performed to generate a global context feature vector for the target ray. :

[0215] ,

[0216] in, Indicates the decoder Transformer,

[0217] This global context feature vector contains macroscopic semantic information about the scene, providing prior guidance for subsequent geometric matching.

[0218] The fifth step is the extraction and fusion of local geometric features.

[0219] Using the global context feature vector as the query vector, the view Transformer aggregates features along the epipolar lines of the source view image and its corresponding first-resolution feature map. Then, the ray Transformer fuses the aggregated features along the epipolar lines with the position and direction information of the target ray along the direction of the target ray to obtain the ray features; including the following steps:

[0220] (1) Epipolar sampling and feature projection: Based on the direction of the target ray, in The image plane from each source viewpoint is sampled along the epipolar line, and features are obtained from the first resolution feature map.

[0221] Ray definition and spatial point sampling: For any target ray to be rendered It originates from the optical center. and direction vector Define the sampling depth range on the light source. ,in The near-end boundary value representing the sampling depth. Represents the far-end boundary value of the sampling depth, within which uniform or stratified sampling is performed. Discrete depth points At this time, the first Three-dimensional spatial sampling points The coordinates are represented as:

[0222] ;

[0223] Coordinate projection transformation: using the pre-acquired first... Camera intrinsic parameter matrix from a single source perspective and extrinsic parameter matrix ,in Let be a rotation matrix. As a position vector, a point in three-dimensional space Projected to the On a two-dimensional image plane from a single source perspective;

[0224] Let the coordinates of the projected two-dimensional pixels be... The homogeneous coordinate transformation formula is:

[0225] ,

[0226] in As a scale factor, and These represent the x and y coordinates of the projection point on the source view image plane, respectively. Through this formula, each three-dimensional point on the target ray is mapped to a two-dimensional projection point on the source view image.

[0227] Feature bilinear interpolation: due to the calculated projected coordinates Typically, the value is a floating-point number, and bilinear interpolation is used to calculate the value from the first... First resolution feature map of the feature map from the source perspective The feature vector corresponding to the sampling point, for each sampling point The features extracted The calculation is as follows:

[0228] ;

[0229] in This represents the bilinear interpolation sampling algorithm.

[0230] (2) Globally guided aggregation: In order to distinguish between occluded areas and real surfaces, the generated global context feature vector is used. As the query vector, the features of the epipolar sampling points obtained from sampling will be used. and its position code As the key and value, input to the Transformer view, where the position is encoded. The contribution weights of different source viewpoints to the target ray are calculated based on the relative geometric positions of the source viewpoint and the target ray, and then dynamically calculated using a self-attention mechanism. The formula is as follows:

[0231] ,

[0232] in Represents a view, such as a Transformer.

[0233] (3) Joint feature embedding: In order to integrate the macro-semantics and micro-geometric details of the scene, the global context feature vector is combined with the micro-geometric details of the scene. With local geometric features The data is concatenated along the channel dimension to form a joint feature embedding. :

[0234] ,in This indicates a splicing operation.

[0235] (4) Obtaining ray features: After embedding the joint features through the ray Transformer, the target ray representation is obtained. :

[0236] ;

[0237] in, This refers to a light ray (Transformer).

[0238] (5) Concentrate the sampling points in the area that contributes the most to the final color;

[0239] First, sample the light evenly. For each sampling point, input the ray Transformer to obtain the original attention weights. This weight reflects the contribution of that depth point to the final color;

[0240] Sigmoid nonlinear smoothing: To address the issue of unstable and noisy dynamic range of the original attention weights, the Sigmoid activation function is used to compress and smooth them, mapping them to the (0,1) interval.

[0241] Probability density function generation: The smoothed discrete weights are fitted with a continuous probability density function along the depth of the light ray using the Gaussian kernel regression method. The calculation formula is:

[0242] ,

[0243] in, Let Gaussian kernel function be defined as:

[0244] ,

[0245] Indicates the target depth position along the target ray. Indicates the first Depth values ​​of uniformly sampled points Represents the natural constant. Pi This is the bandwidth parameter, used to control the smoothness of the probability density function;

[0246] Fine sampling: Utilizing the inverse transform sampling algorithm to additionally sample data from the probability density function. These are fine sample points that are automatically clustered on the surface of the object, i.e., the areas with high weight.

[0247] Final Feature Aggregation: The coarse and fine sampling points are merged and input into the ray Transformer again for multi-layer self-attention aggregation, outputting the final target ray features. .

[0248] Step 6, Generation of the new perspective image: The final ray features output from the ray Transformer are input into the multilayer perceptron decoder to synthesize the new perspective image; the final target ray features are then processed... Input a multilayer perceptron decoder The RGB color value corresponding to the target ray is obtained by mapping. :

[0249] ,

[0250] The calculated RGB color values ​​of each target ray Arrange the pixel coordinates of the new perspective to be synthesized to generate a complete new perspective image.

[0251] In practical applications, model training and optimization can be enhanced by constructing a loss function and using the backpropagation algorithm to update network parameters.

[0252] After the neural network model is built, it needs to be trained end-to-end using real images from known viewpoints. The specific steps are as follows:

[0253] (1) Loss Function Definition: In this embodiment, photometric loss is used as the supervision signal. For each ray in the training batch... Calculate its predicted color value Compared with the actual observed color value The mean squared error (MSE) between them is calculated as follows:

[0254]

[0255] in, This is the set of rays sampled in each iteration.

[0256] (2) Parameter Optimization: The Adam optimizer is used to update all learnable parameters in the network (including the weights of the hierarchical bidirectional feature aggregation network, Mamba module, Transformer module, and multi-perceptron decoder). In this embodiment, the initial learning rate of the multi-scale feature extraction module is set to... The initial learning rate for the remaining modules is The learning rate is adjusted using an exponential decay strategy. The training process continues until the loss function converges.

[0257] Table 1. Comparison of the method of this invention with the state-of-the-art novel perspective synthesis method on the LLFF dataset.

[0258]

[0259] As shown in Table 1, it can be seen that the method of the present invention outperforms other comparative methods in core indicators such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual similarity (LPIPS) under the stringent condition of only inputting 2 to 3 sparse viewpoints.

[0260] Experimental results fully demonstrate that this invention, through the synergistic effect of hierarchical feature extraction, Mamba global context modeling and local geometric features, and in conjunction with an adaptive sampling strategy based on Sigmoid smoothing, can effectively overcome the problem of insufficient geometric constraints in low-sample scenes, and significantly improve the rendering quality and generalization ability in complex textures, reflective and occluded areas.

[0261] The method described in this invention can be implemented on a computer device including a processor and memory. Hardware configuration: The experiments and tests in this embodiment were completed on a high-performance computing platform, specifically configured as follows: a central processing unit (CPU) of Intel Xeon series, a graphics processing unit (GPU) of NVIDIA RTX 4090 (24GB VRAM), and 64GB of RAM. Software environment: The algorithm is implemented based on the Python programming language and the PyTorch deep learning framework.

[0262] like Figure 4 As shown, this invention provides comparison results between synthesized new perspective images and corresponding real images in five different complex scenes. The left column shows the new perspective images synthesized by the method of this invention, and the right column shows the corresponding real images. The comparison shows that the images generated by the method of this invention under limited sample conditions exhibit a high degree of consistency with real images in terms of geometric structure, texture details, lighting representation, and material realism, fully verifying the effectiveness of this invention.

[0263] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination, characterized in that, Includes the following steps: 11) Dataset construction and sparse input acquisition: Obtain a sparse source view image set of the scene to be reconstructed and the camera pose parameters corresponding to each image; 12) Define a hierarchical bidirectional feature aggregation network: Define a hierarchical bidirectional feature aggregation network including an encoder for feature extraction, a multi-scale self-attention module for cross-scale feature enhancement, and a decoder for fine features; 13) Extraction of multi-scale features; 14) Global context modeling; 15) Extraction and fusion of local geometric features; 16) Generation of new perspective images: Input the final ray features output by the ray Transformer into the multilayer perceptron decoder to synthesize new perspective images; The final target light characteristics Input a multilayer perceptron decoder The RGB color value corresponding to the target ray is obtained by mapping. : ， The calculated RGB color values ​​of each target ray Arrange the pixel coordinates of the new perspective to be synthesized to generate a complete new perspective image.

2. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 1, characterized in that, In the dataset construction and sparse input acquisition steps, for the target new viewpoint to be synthesized, its spatial distance with all source viewpoints is calculated, and the closest viewpoint is selected from the sparse source viewpoint image set. The image is used as the source view image, and the source view image and the new target view image are combined to form a source-target view pair to establish the relative geometric projection relationship between the source view and the target view. Includes the following steps: 21) Acquire multiple source view images of the scene to be reconstructed, and pre-calculate the camera intrinsic and extrinsic parameters of each image using the structure-in-motion algorithm. The intrinsic parameters include focal length and principal point, and the extrinsic parameters include rotation matrix and position vector. For static backgrounds and dynamic objects in the images, size normalization is uniformly performed. 22) For each new viewpoint to be synthesized, calculate its spatial Euclidean distance to all source viewpoints: Based on pre-calculated camera extrinsics, obtain the position vector of the target from a new perspective. With the Position vector from the source perspective The position vector represents the three-dimensional coordinates of the camera's optical center in the world coordinate system. Spatial Euclidean distance between the two The calculation formula is: ， in, , , Represent the target's new perspective position vector, respectively. In the world coordinate system axis, shaft and Spatial coordinate components of the axis, , , They represent the first Individual source perspective position vector In the world coordinate system axis, shaft and Spatial coordinate components of the axis, It is a three-dimensional real vector space; Based on the calculated spatial Euclidean distance Sort all source views in ascending order and select the top ones. Each corresponding image constructs a source-target view pair.

3. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 1, characterized in that, The extraction of multi-scale features involves: using a hierarchical bidirectional feature aggregation network to extract features from the source viewpoint image, generating a multi-scale feature map for each source viewpoint image. The multi-scale feature map is divided into a first-resolution feature map and a second-resolution feature map; this includes the following steps: 31) Input the source view image into the encoder. The encoder consists of stacked convolutional neural network modules forming the basic feature extraction unit. The processing flow of the convolutional neural network module can be formally represented as follows: ， Where Input represents the input, Conv represents the convolutional layer, BN represents the batch normalization layer, and ReLU is the activation function. Represents output characteristics; Based on the convolutional neural network module, a cascaded architecture is used to extract multi-scale features step by step. The specific process is as follows: The first stage generates C1: the source view image, with a size of [size missing]. , Image height, The image width is first downsampled, then three stacked convolutional neural network modules are used for feature extraction to generate a first-stage feature map C1 with a resolution one-quarter of the original image and a size of [size missing]. ; The second stage generates C2: The first-stage feature map C1 is used as input for further downsampling, and then further abstracted features are extracted using four stacked convolutional neural network modules, generating a second-stage feature map C2 with a resolution one-eighth that of the original image, and a size of [size missing]. ; The third stage generates C3: The second-stage feature map C2 is used as input, downsampled again, and deep feature extraction is performed through six stacked convolutional neural network modules to generate a third-stage feature map C3 with a resolution of one-sixteenth of the original image. The size is... ; The feature maps {C1, C2, C3} generated in the above three stages together constitute the multi-scale coding feature pyramid of the encoder; 32) Cross-scale feature enhancement: A multi-scale self-attention module is used to perform cross-scale interaction on the encoded feature maps at different stages of the encoder. The multi-scale self-attention module downsamples the three-stage feature maps generated by the encoder to a uniform size and generates a query matrix through linear transformation. Key matrix Sum matrix , To explore the potential positional relationships and long-distance dependencies between objects at different scales, a self-attention mechanism is applied to calculate the weights, as shown in the following formula: ， ， ， ， in, and Feature maps representing different scales. This is the weight matrix. For scaling parameters, The generated self-attention feature map; 33) Define the core processing components in the decoder: The decoder contains an information integration attention module and a global and local feature co-fusion module, which work together to restore feature resolution and generate multi-scale feature maps. 34) Cascaded decoding and output of multi-scale feature maps: 341) Construct an information integration attention module; 342) Constructing a local feature collaborative fusion module 343) Generate the second resolution feature map: The feature map output by the multi-scale self-attention module is processed by the global and local feature fusion module and then upsampled. It is then concatenated with the second-stage feature map C2 and processed by the information integration attention module and the global and local feature fusion module in sequence. Finally, the second resolution feature map is output through 1×1 convolution. 344) Generate the first resolution feature map: After upsampling the second resolution feature map, it is concatenated with the first stage feature map C1, and then processed by the information integration attention module and the global and local feature fusion module in sequence. Finally, the first resolution feature map is output through 1×1 convolution.

4. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 1, characterized in that, The global context modeling involves: using the multi-axis sparse state space Mamba module to perform sequence modeling on the second-resolution feature map, constructing a global context feature vector containing cross-view dependencies; including the following steps: 41) Define the adaptive size division algorithm: For the input second-resolution feature map, its size is , For feature map height, Set the preset ideal partition size for the feature map width. , This indicates the preset division height. Indicates the preset partition width, and checks the feature map height. Is it based on the current preset division height? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ; Check feature map width Is it subject to the current preset partition width? If divisible, then perform a decrement operation. until satisfaction is found The largest integer ; The final result This refers to an adaptive partitioning size without fill; like or If reducing the size to 1 still fails to satisfy the integer division condition, then the corresponding block size or grid size is directly set to 1, at which point it degenerates into pixel-by-pixel processing; 42) Set up local window state space scanning unit processing: Calculate the adaptive block size using an adaptive size partitioning algorithm , This indicates the preset block height. This indicates the preset block width, and the feature map is reshaped and transposed. Shape tensors, Indicates the batch size of the input data. For feature map height, The width of the feature map. This represents the number of channel dimensions in the feature map, maintaining independence between blocks, and assigning a size of [missing information]. The local blocks are flattened into a one-dimensional sequence, and the Mamba state space model is independently applied for scanning to capture the pixel dependencies within the local region. 43) Set global sparse grid state space scan cell processing: Calculate the adaptive mesh size using an adaptive size partitioning algorithm , This indicates the preset grid height. This indicates the preset grid width, which reshapes the feature map into... Shape tensors, Indicates the batch size of the input data. This indicates the number of channel dimensions in the feature map. For feature map height, The feature map width is used as the basis for dimensional permutation, which aggregates pixels belonging to the same grid position. Extract grid point sequences with extremely large spans and lengths of . Input state-space model; 44) Configure cross-view collaborative state space scan unit processing: Will The feature maps from each source viewpoint are stitched together along the viewpoint dimension to form a shape of... The tensor will Considering the sequence length, for each pixel at a spatial location, a state space model is applied to scan along the view axis to capture the geometric consistency and semantic association between different viewpoints; 45) Global Feature Generation: After the second-resolution feature map is processed through steps 42), 43), and 44), an implicit scene representation is obtained. Using the Transformer decoder, the target ray is... Position and orientation codes are used as queries in The search is performed to generate a global context feature vector for the target ray. : ， in, Indicates the decoder Transformer, This global context feature vector contains macroscopic semantic information about the scene, providing prior guidance for subsequent geometric matching.

5. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 1, characterized in that, The extraction and fusion of local geometric features are as follows: using the global context feature vector as the query vector, the view Transformer aggregates features on the epipolar lines of the source view image and its corresponding first resolution feature map, and then the ray Transformer fuses the aggregated features on the epipolar lines with the position and direction information of the target ray along the direction of the target ray to obtain ray features; including the following steps: 51) Epipolar sampling and feature projection: Based on the direction of the target ray, in Sampling is performed along the epipolar line on the image plane of each source viewpoint, and features are obtained from the first resolution feature map; 52) Globally Guided Aggregation: To distinguish between occluded regions and the real surface, the global context feature vector generated in step 13) is used. As the query vector, the features of the epipolar sampling points obtained from sampling will be used. and its position code As the key and value, input to the Transformer view, where the position is encoded. The contribution weights of different source viewpoints to the target ray are calculated based on the relative geometric positions of the source viewpoint and the target ray, and then dynamically calculated using a self-attention mechanism. The formula is as follows: ， in Represents a view, such as a Transformer. 53) Joint Feature Embedding: In order to integrate the macroscopic semantics and microscopic geometric details of the scene, the global context feature vector is... With local geometric features The data is concatenated along the channel dimension to form a joint feature embedding. : ,in Indicates a splicing operation; 54) Obtaining ray features: After embedding the joint features through a ray Transformer, the target ray representation is obtained. : ； in, Represents a light ray Transformer; 55) Concentrate sampling points in the region that contributes the most to the final color; 551) First, sample the light uniformly. For each sampling point, input the ray Transformer to obtain the original attention weights. This weight reflects the contribution of that depth point to the final color; 552) Sigmoid Nonlinear Smoothing: To address the problem of unstable and noisy dynamic range of the original attention weights, the Sigmoid activation function is used to compress and smooth them, mapping them to the (0,1) interval. 553) Probability density function generation: The smoothed discrete weights are fitted to a continuous probability density function along the depth of the light ray using the Gaussian kernel regression method. The calculation formula is: ， in, Let Gaussian kernel function be defined as: ， Indicates the target depth position along the target ray. Indicates the first Depth values ​​of uniformly sampled points Represents the natural constant. Pi This is the bandwidth parameter, used to control the smoothness of the probability density function; 554) Fine sampling: Using the inverse transform sampling algorithm to additionally sample data from the probability density function. These are fine sample points that are automatically clustered on the surface of the object, i.e., the areas with high weight. 555) Final Feature Aggregation: The coarse and fine sampling points are merged and input into the ray Transformer again for multi-layer self-attention aggregation, outputting the final target ray features. .

6. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 3, characterized in that, The construction of the information integration attention module includes the following steps: 61) Two feature maps of the same resolution from the encoder and decoder are concatenated along the channel dimension to form a combined feature. ; 62) respectively for At height and width Average pooling and max pooling are performed in the direction to generate channels. eigenvectors on , and ; Through concat operation and two-dimensional convolution... One-dimensional convolution Batch normalized normalization (BN) and sigmoid activation function processing are used to obtain spatial attention weights in the height and width dimensions. and : ， ； 63) Combine the generated weights with the combined features Multiplying and concatenating the results yields the final output of the information integration attention module: ， Among the symbols Indicates element-wise multiplication, symbol This indicates element-wise addition.

7. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 3, characterized in that, The construction of the global and local feature collaborative fusion module includes the following steps: 71) Setting up local information branches: Perform the following steps on the input feature maps respectively: Convolution and Convolution is performed, and the results of the two processes are summed to form initial local features; subsequently, cascaded dilated convolution processing is performed: a set of convolution dilation rates is defined. The feature maps are processed iteratively multiple times using depthwise separable convolutions, with each iteration employing a different dilation rate to abstract and represent objects of different sizes in various directions. Finally, all intermediate feature maps generated during the concatenation process are concatenated along the channel dimension and processed through a single... Convolution adjusts the number of channels to obtain the output of the local information branch. ; 72) Define a global information branch: Use the state-space model as the main body to extract global features and model long-range contextual information. First, the input feature map is uniformly divided into four sub-feature blocks along the channel dimension, denoted as... Secondly, each sub-feature block is input into the visual state space module for parallel processing, utilizing the selective scanning mechanism of the visual state space module to capture global dependencies. Finally, the four processed sub-feature blocks are reassembled along the channel dimension to obtain the output of the global information branch. ; 73) Setting the feature embedding head: To effectively embed local information into global information, a feature similarity-based fusion mechanism is designed, specifically as follows: First, output the local branch. With global branch output By performing element-by-element addition, preliminary fusion characteristics are obtained. ; Secondly, calculation and Cosine similarity between The formula is as follows: ， in, and These represent the pixel values ​​at the same location in the two feature maps. Total number of pixels For pixel index; Next, using the Sigmoid function to... Perform nonlinear mapping to generate weight masks; perform preliminary feature fusion. conduct Convolution yields intermediate features ;Will The weighted features are obtained by multiplying them element-wise with the weight mask. ; Finally, With the process After convolution By adding elements one by one, the residual characteristics are obtained. Interpolation and upsampling are performed to output the final enhanced feature map. Its computational logic is expressed as follows: ， ， ， ， Among them, symbols Indicates element-wise multiplication, symbol This indicates element-wise addition. express convolution, This indicates an interpolation upsampling operation.

8. The method for synthesizing novel perspective images with few samples based on multi-scale hybrid sensing and state-space coordination according to claim 5, characterized in that, The epipolar sampling and feature projection include the following steps: 81) Ray definition and spatial point sampling: For any target ray to be rendered It originates from the optical center. and direction vector Define the sampling depth range on the light source. ,in The near-end boundary value representing the sampling depth. Represents the far-end boundary value of the sampling depth, within which uniform or stratified sampling is performed. Discrete depth points At this time, the first Three-dimensional spatial sampling points The coordinates are represented as: ； 82) Coordinate projection transformation: using the pre-acquired first... Camera intrinsic parameter matrix from a single source perspective and extrinsic parameter matrix ,in Let be a rotation matrix. As a position vector, a point in three-dimensional space Projected to the On a two-dimensional image plane from a single source perspective; Let the coordinates of the projected two-dimensional pixels be... The homogeneous coordinate transformation formula is: ， in As a scale factor, and These represent the x and y coordinates of the projection point on the source view image plane, respectively. Through this formula, each three-dimensional point on the target ray is mapped to a two-dimensional projection point on the source view image. 83) Characteristic bilinear interpolation: Due to the calculated projected coordinates Typically, the value is a floating-point number, and bilinear interpolation is used to calculate the value from the first... First resolution feature map of the feature map from the source perspective The feature vector corresponding to the sampling point, for each sampling point The features extracted The calculation is as follows: ； in This represents the bilinear interpolation sampling algorithm.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, enables the method for synthesizing novel perspective images based on multi-scale hybrid sensing and state space collaboration, as described in any one of claims 1-8.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can implement the method for synthesizing novel viewpoint images based on multi-scale hybrid sensing and state space collaboration as described in any one of claims 1-8.

Images