Traditional picture multi-view generation method combined with bidirectional state space model

Abstract

The invention belongs to the technical field of computer vision, and relates to a traditional picture multi-view generation method combined with a bidirectional state space model, which comprises the following steps of: acquiring a traditional picture image, respectively executing low-resolution global estimation and high-resolution local estimation on the traditional picture image, and fusing estimation results to generate a depth map; obtaining a corresponding image pair, and performing damage repair on the obtained image pair to obtain a new view angle input pair; the new view angle input pair is used as training data, a traditional picture multi-view angle generation model is trained, and the traditional picture multi-view angle generation model comprises a multi-modal feature coding module, an improved bidirectional state space model and a multi-plane generation module; inputting the tested landscape painting image and the corresponding depth map into the trained traditional painting multi-view generation model, and outputting a multi-view landscape painting image; according to the method, the visual coherence and detail expressive force of multi-view synthesis are remarkably improved while the traditional painting artistic conception is kept.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for generating multiple perspectives of a traditional image by combining a bidirectional state space model. Background Art

[0002] With the rapid development of digital technology, the demand for three-dimensional image representation is growing in fields such as intelligent vision systems. New perspective synthesis methods based on neural rendering have made significant progress in 3D reconstruction, virtual reality, and other fields. Multi-plane imagery (MPI), as an effective scene representation method, demonstrates significant advantages in occlusion handling, rendering efficiency, and end-to-end learning, and is widely used for perspective synthesis tasks in real-world images.

[0003] However, traditional Chinese paintings use the scattered perspective method, and there is a lack of clear spatial depth correspondence in the picture. During the perspective change process, the originally visible picture may be blocked by other areas. Due to the existence of blocked areas, the image often appears hollow or broken. Traditional image restoration methods are difficult to handle the long-range dependency of scattered perspective, and cannot maintain the continuity of ink strokes and the natural transition of virtual and real layers when the perspective changes. It cannot maintain the consistency of the original brushstrokes and the consistency of the artistic style. As a result, the existing multi-perspective synthesis methods have problems such as structural fracture, occlusion restoration distortion, and artistic style inaccuracy, making it difficult to directly migrate and apply them.

[0004] Therefore, it is necessary to design a multi-perspective image generation method for traditional Chinese paintings that improves the above problems. Summary of the Invention

[0005] To address the problems of the prior art, the present invention provides a method for generating multiple perspectives of traditional paintings by combining a bidirectional state space model, which uses a two-stage collaborative architecture to achieve new perspective synthesis of ancient paintings. The method includes the following steps:

[0006] Step 1: Obtain traditional painting image I s , for traditional painting images I s Perform low-resolution global estimation and high-resolution local estimation respectively, and fuse the estimation results to generate the depth map Ds, and obtain the corresponding image pair (I s , D s );

[0007] Step 2: The image pair (I s , D s ) to repair the damage and obtain the new perspective image pair (I t , D t );

[0008] Step 3: Input the new perspective into (I t , D t) as training data to train the traditional painting multi-view generation model;

[0009] The traditional multi-view generation model of the painting includes at least an improved bidirectional state space model and a multi-plane generation module;

[0010] The improved two-way state space model uses the two-way state space model to analyze the new perspective input pair (I t , D t ) performs forward and reverse convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations to capture long-range dependencies in scattered perspectives; a multi-plane generation module is used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r Adopting the "soft distribution" strategy to simulate the virtual-real transition of traditional painting, outputting the color and density of each plane;

[0011] Step 4: Input the test traditional painting image and its corresponding depth map into the trained traditional painting multi-view generation model, and output the multi-view traditional painting image.

[0012] Further, the damage repair method in step 2 includes: s , D s ) The image pair is input into the virtual camera, and the virtual camera pose is transformed Generate intermediate damage view (I m ′,D m ′), using the disparity mask M d Identify the occluded area, combine the ink line structure prior and depth information to repair the damaged area, and obtain the composite edge structure E c ; Through a dual-branch network Processing color and depth information, while using PatchGAN as a discriminator, local perception of the image, and evaluation of edge structure and texture details, the expression is:

[0013]

[0014] in, Represents the connected composite edge structure E c and the disparity mask M d ⊙ represents the fusion of real edge content and generated edge content to obtain a new perspective input pair (I t , D t ).

[0015] Furthermore, the traditional multi-view generation model in step 3 also includes a multimodal feature encoding module for extracting new perspective input pairs (I t , D t) Global structural features, establishing physical associations between color and geometry;

[0016] First, the new perspective input pair (I t , D t ) is downsampled to generate an initial depth plane set Then the new perspective is input to (I t , D t ) and the depth plane set Channel splicing is performed to form multi-channel input features, which are fed into a five-layer encoder-pooling architecture to extract global structural features T 0 , establish the physical association between image color and geometry, the expression is:

[0017]

[0018] in represents the s-th layer coding module, represents the s-th layer downsampling module, Represents the cascade operation of five layers of encoding-pooling modules.

[0019] Furthermore, the improved two-way state space model is used to analyze the new perspective input pair (I t , D t ) performs forward and backward convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations and capture long-range dependencies in scattered perspectives, specifically including:

[0020] Global structural feature T 0 Perform linear mapping to generate bidirectional feature bases x and z. For the feature base x, perform forward and reverse processing on it to obtain features in two directions, and recursively update the hidden state.

[0021]

[0022]

[0023] in, represents the forward one-dimensional convolution, To flip the sequence x and then perform the convolution operation;

[0024] The feature base z is activated by the SiLU function to obtain z′, which is combined with the bidirectional features and the global structure and overall content features of the image to obtain the global temporal information T of the image:

[0025]

[0026] A multi-head attention mechanism is used for T to focus on key areas and optimize depth calibration.

[0027] Furthermore, a multi-plane generation module is used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r A "soft distribution" strategy is used to simulate the virtual-real transition of traditional paintings, outputting the color and density of each plane. Specifically, it includes:

[0028] The new perspective obtained in step 2 is input to (I t , D t ) is spliced ​​with the global temporal information T along the channel dimension and input into the ResMamba feature extractor. The ResMamba feature extractor includes a feature extraction module and a bidirectional state space model connected in sequence to extract multi-scale features F, model the dependency between different image regions, and use a multi-plane feature mask to divide it into feature masks M i , background mask and foreground mask Through the decoder D r The "soft allocation" strategy is used to simulate the virtual-real transition of traditional paintings and output the color c of each plane. i and density σ i , the expression is:

[0029]

[0030] in, Represents the connected composite edge structure E c and the disparity mask M d ⊙ represents the fusion of real edge content and generated edge content to obtain a new perspective input pair (I t , D t ).

[0031] Furthermore, the method adopts a joint loss function in the process of training the multi-view synthesis model, including: pixel-level error metric loss, structural similarity loss, perceptual loss, focal frequency loss and depth regularization loss;

[0032] The depth regularization loss includes depth sorting loss and depth distribution loss, which are used to ensure the reasonable arrangement of depth values ​​and close to the actual depth distribution;

[0033] The pixel-level error metric loss is used to measure the pixel-level error between the generated image and the real image, constraining the pixel-level accuracy so that the generated image retains the artistic style and details of the original as much as possible. The expression is:

[0034]

[0035] in, Indicates the generated image, Represents the input image pair, H and W represent the height and width of the image respectively; i, j is a pixel at a specific position in the image;

[0036] The perceptual loss extracts high-level semantic features through a pre-trained convolutional neural network and compares the differences between the generated image and the target image in the feature space, thereby capturing the brushstrokes, texture, and style information in the painting. The expression is:

[0037]

[0038] Among them, φ l (x) represents the feature extractor output of the l-th layer of the convolutional neural network, N l represents the total number of elements in the feature map of the lth layer, and L represents the number of selected feature layers;

[0039] Structural similarity loss enhances the perceptual quality of the image by evaluating the consistency of brightness, contrast, and structural information between the generated image and the target image in a local area. The expression is:

[0040]

[0041] Among them, μ (·) represents the local mean of the image, represents the local covariance between the two, represents the local variance of the image, C (·) is a small constant used to avoid zero denominator and improve numerical stability;

[0042] Focusing on frequency loss, by analyzing the frequency domain distribution of the image, we can enhance the recovery of high-frequency components while protecting the continuity of low-frequency components. The expression is:

[0043]

[0044] Where F(x) represents the Fourier transform of the image, α>0 and β>0 are hyperparameters that control the degree of attention to different frequencies, and i represents the index in the frequency domain;

[0045] At the same time, the depth regularization loss,

[0046] Used to constrain the physical rationality of the relationship between MPI layers and network output, ensure that the prediction value of the traditional multi-view generation model meets the actual requirements, and add regularization loss The expression is as follows:

[0047]

[0048] The depth sorting loss Used to ensure that the depth values ​​are arranged in a reasonable order to avoid unreasonable jumps or intersections; depth allocation loss Used to make the predicted depth value closer to the actual depth distribution D s , thereby improving geometric consistency, the expression is:

[0049]

[0050] Among them, d i Indicates the depth of the i-th layer.

[0051] Furthermore, the joint loss function also includes a smoothing loss, which is used to ensure the depth continuity of the distant part and prevent the depth prediction from jumping violently;

[0052] Smoothing loss The expression is:

[0053]

[0054] in, represents the gradient of the depth map, represents the gradient of the color image, Used to dynamically adjust the smoothing intensity based on the color gradient;

[0055] Beneficial effects of the present invention:

[0056] The present invention combines the traditional painting multi-perspective generation method of the bidirectional state space model, performs denoising, background unification and edge enhancement preprocessing on the input landscape painting to obtain the traditional painting image, and realizes high-quality multi-perspective landscape painting image synthesis through depth estimation, damage repair, bidirectional state space modeling and multi-plane image optimization, which solves the problems of structural fracture, occlusion repair distortion and artistic style inaccuracy in multi-perspective synthesis caused by the scattered perspective characteristics of traditional landscape painting. While maintaining the artistic conception of traditional painting, it significantly improves the visual coherence and detail expression of multi-perspective synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the method for generating multiple perspectives of a traditional painting in combination with a bidirectional state space model according to the present invention;

[0058] Figure 2 It is to compare the edge coherence of the damaged view and the repair result;

[0059] Figure 3 It is the overall framework diagram of the present invention;

[0060] Figure 4 It is a structural diagram of the improved bidirectional state space model of the present invention;

[0061] Figure 5This is a structural diagram of the ResMamba feature extractor of the present invention;

[0062] Figure 6 It is a qualitative comparative analysis of ablation experiments on different components using the method of the present invention;

[0063] Figure 7 This is a qualitative comparative analysis of the ablation experiments conducted by the method of the present invention on different components of the joint loss function;

[0064] Figure 8 This is a qualitative comparative analysis of the landscape painting generation results of the method of the present invention and other advanced methods;

[0065] Figure 9 This is a display of the multi-view generation results for a single image by the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] refer to Figure 1 The present invention provides a method for generating multi-views of traditional paintings in combination with a bidirectional state space model, comprising the following steps: Step 1: Obtaining a traditional painting image I s , for traditional painting images I s Perform low-resolution global estimation and high-resolution local estimation respectively, and fuse the estimation results to generate a depth map D s , get the corresponding image pair (I s , D s );

[0068] It should be noted that 7,324 landscape paintings from different dynasties were subjected to denoising, background unification and edge enhancement preprocessing to obtain image I s Dataset, depth map D corresponding to these images s The dataset is generated by the method proposed in this paper. It is divided into a training set (14,448 images, including 7,224 ancient paintings and their corresponding 7,224 depth maps) and a test set (200 images, including 100 ancient paintings and their corresponding 100 depth images).

[0069] Step 2: The image pair (I s , D s ) to repair the damage and obtain the new perspective image pair (I t , D t );

[0070] Step 3: Input the new perspective into (I t , D t ) as training data to train the traditional painting multi-view generation model;

[0071] Specifically, the entire process framework of the multi-view generation model of traditional Chinese paintings is as follows: Figure 3 As shown in the figure, the input data is firstly transformed into a multimodal feature encoding module and an improved bidirectional state space model to achieve dynamic optimization of the depth distribution of ancient paintings:

[0072] The traditional multi-view generation model of the painting includes at least an improved bidirectional state space model and a multi-plane generation module;

[0073] The improved two-way state space model uses the two-way state space model to analyze the new perspective input pair (I t , D t ) performs forward and reverse convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations to capture long-range dependencies in scattered perspectives; a multi-plane generation module is used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r Adopting the "soft distribution" strategy to simulate the virtual-real transition of traditional painting, outputting the color and density of each plane;

[0074] Step 4: Input the test traditional painting image and its corresponding depth map into the trained traditional painting multi-view generation model, and output the multi-view traditional painting image;

[0075] Furthermore, the depth map D in step 1 s The generation of is achieved through a high- and low-resolution complementary depth estimation network, specifically including:

[0076] For image I s Perform low-resolution global estimation and high-resolution local estimation respectively, and fuse details and structures through the deep fusion network to obtain the depth map D s ;

[0077]

[0078] where f LR (·) and f HR (·) are low-resolution global estimation network and high-resolution local estimation network respectively, It is a deep fusion network based on the Pix2pix architecture.

[0079] Further, the damage repair method in step 2 includes: s , Ds ) The image pair is input into the virtual camera, and the virtual camera pose is transformed Generate intermediate damage view (I m ′,D m ′), using the disparity mask M d Identify the occluded area, combine the ink line structure prior and depth information to repair the damaged area, and obtain the composite edge structure E c ; Through a dual-branch network Processing color and depth information, while using PatchGAN as a discriminator, local perception of the image, and evaluation of edge structure and texture details, the expression is:

[0080]

[0081] in, Represents the connected composite edge structure E c and the disparity mask M d ⊙ represents the fusion of real edge content and generated edge content to obtain a new perspective input pair (I t , D t ).

[0082] PatchGAN is used as a discriminator to perform local perception on the image and evaluate edge structure and texture details, such as Figure 2 As shown, the last column is the repair result for the damaged view in the middle of the second column, which maintains the consistency of the depth map and image content while repairing the texture;

[0083] Furthermore, the traditional multi-view generation model in step 3 also includes a multimodal feature encoding module for extracting new perspective input pairs (I t , D t ) Global structural features, establishing physical associations between color and geometry;

[0084] First, the new perspective input pair (I t , D t ) is downsampled to generate an initial depth plane set Then, the new perspective is input to (I t , D t ) and the depth plane set Channel splicing is performed to form multi-channel input features, which are fed into a five-layer encoder-pooling architecture to extract global structural features T 0 , establish the physical association between image color and geometry, the expression is:

[0085]

[0086] in represents the s-th layer coding module, represents the s-th layer downsampling module, Represents the cascade operation of five layers of encoding-pooling modules.

[0087] Furthermore, the improved two-way state space model is used to analyze the new perspective input pair (I t , D t ) performs forward and backward convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations and capture long-range dependencies in scattered perspectives, specifically including:

[0088] Global structural feature T 0 Perform linear mapping to generate bidirectional feature bases x and z. For the feature base x, perform forward and reverse processing on it to obtain features in two directions, and recursively update the hidden state.

[0089]

[0090]

[0091] in, represents the forward one-dimensional convolution, To flip the sequence x and then perform the convolution operation;

[0092] The feature base z is activated by the SiLU function to obtain z′, which is combined with the bidirectional features and the global structure and overall content features of the image to obtain the global temporal information T of the image:

[0093]

[0094] A multi-head attention mechanism is used for T to focus on key areas and optimize depth calibration.

[0095] Furthermore, a multi-plane generation module is used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r A "soft distribution" strategy is used to simulate the virtual-real transition of traditional paintings, outputting the color and density of each plane. Specifically, it includes:

[0096] The new perspective obtained in step 2 is input to (I t , D t ) is spliced ​​with the global temporal information T along the channel dimension and input into the ResMamba feature extractor. The ResMamba feature extractor includes a feature extraction module and a bidirectional state space model connected in sequence to extract multi-scale features F, model the dependency between different image regions, and use a multi-plane feature mask to divide it into feature masks Mi , background mask and foreground mask Through the decoder D r The "soft allocation" strategy is used to simulate the virtual-real transition of traditional paintings and output the color c of each plane. i and density σ i , the expression is:

[0097]

[0098] It should be noted that this application designs a ResMamba feature extractor, the specific structure of which is as follows Figure 5 As shown in the figure, the ResMamba feature extractor fuses multi-scale information and uses the "soft assignment" of foreground / background masks to simulate the virtual-real transition of traditional paintings, achieving an effective conversion of images and their depth information into multi-scale information. The feature extractor captures rich features from low-level details such as texture and edges to high-level semantics such as overall composition and spatial relationships, and uses them to guide the decoder to predict and generate the color and density of each plane, highlighting the subject and de-emphasizing the background.

[0099] Specifically, the new perspective obtained in step 2 is input to (I t , D t ) is spliced ​​with the global temporal information T along the channel dimension, and depth clues are introduced by display, and the physical association between image color and geometric structure is established, which provides structured constraints for learning subtle relationships between multiple planes and forms a bidirectional state space model improved by structured multimodal input. Finally, the obtained multi-scale features F are used to model the dependencies between different picture areas, promote the fusion of cross-plane information, and integrate the information of each area of ​​the image, thereby forming a more comprehensive global perspective.

[0100] Input the new perspective obtained in step 2 into (I t , D t ) After multi-plane feature masking, the feature mask in traditional Chinese calligraphy and painting is extracted and divided into feature mask M i , background mask Foreground mask Three parts.

[0101] Mask the background The shared feature map F obtained by multi-scale feature extraction is multiplied to extract the context information related to the i-th plane, so as to better repair the occluded area and restore the blurred parts due to occlusion or overlapping brushstrokes. In traditional ancient paintings, the layers are often not rigidly divided, but are naturally transitioned by fading, leaving white space, etc. Therefore, the extracted context information is combined with the feature mask M iStitching can "softly" assign visible pixels to specific planes, imitating this technique so that each plane can be smoothly connected and clearly layered when synthesized. Finally, the spliced ​​features are input to the decoder In the foreground mask By filtering the background information and finally outputting the color and density of the plane, the generated painting is like the ancient painters who deliberately highlighted the subject and downplayed the background, making the subject of the picture more expressive while the overall artistic conception is not disturbed by the messy background information.

[0102] Furthermore, the method adopts a joint loss function in the process of training the multi-view synthesis model, including:

[0103] Pixel-level error metric loss, structural similarity loss, perceptual loss, focal frequency loss, and depth regularization loss;

[0104] The depth regularization loss includes depth sorting loss and depth distribution loss, which are used to ensure the reasonable arrangement of depth values ​​and close to the actual depth distribution;

[0105] The pixel-level error metric loss is used to measure the pixel-level error between the generated image and the real image, constraining the pixel-level accuracy so that the generated image retains the artistic style and details of the original as much as possible. The expression is:

[0106]

[0107] in, Indicates the generated image, Represents the input pair image, H and W represent the height and width of the image respectively;

[0108] The perceptual loss extracts high-level semantic features through a pre-trained convolutional neural network and compares the differences between the generated image and the target image in the feature space, thereby capturing the brushstrokes, texture, and style information in the painting. The expression is:

[0109]

[0110] Among them, φ l (x) represents the feature extractor output of the l-th layer of the convolutional neural network, N l represents the total number of elements in the feature map of the lth layer, and L represents the number of selected feature layers;

[0111] Structural similarity loss enhances the perceptual quality of the image by evaluating the consistency of brightness, contrast, and structural information between the generated image and the target image in a local area. The expression is:

[0112]

[0113] Among them, μ(·) represents the local mean of the image, represents the local covariance between the two, represents the local variance of the image, C (·) is a small constant used to avoid zero denominator and improve numerical stability;

[0114] Focusing on frequency loss, by analyzing the frequency domain distribution of the image, we can enhance the recovery of high-frequency components while protecting the continuity of low-frequency components. The expression is:

[0115]

[0116] Where F(x) represents the Fourier transform of the image, α>0 and β>0 are hyperparameters that control the degree of attention to different frequencies, and i represents the index in the frequency domain;

[0117] At the same time, the depth regularization loss,

[0118] Used to constrain the physical rationality of the relationship between MPI layers and network output, ensure that the prediction value of the traditional multi-view generation model meets the actual requirements, and add regularization loss The expression is as follows:

[0119]

[0120] The depth sorting loss Used to ensure that the depth values ​​are arranged in a reasonable order to avoid unreasonable jumps or intersections; depth allocation loss Used to make the predicted depth value closer to the actual depth distribution D s , thereby improving geometric consistency, the expression is:

[0121]

[0122] Among them, d i Indicates the depth of the i-th layer.

[0123] Furthermore, the joint loss function also includes a smoothing loss, which is used to ensure the depth continuity of the distant part and prevent the depth prediction from jumping violently;

[0124] Smoothing loss The expression is:

[0125]

[0126] in, represents the gradient of the depth map, represents the gradient of the color image, Used to dynamically adjust the smoothing intensity based on the color gradient;

[0127] It is worth noting that the present invention combines the traditional painting multi-perspective generation method of the bidirectional state space model, performs denoising, background unification and edge enhancement pre-processing on the input landscape painting to obtain the traditional painting image, and realizes high-quality multi-perspective landscape painting image synthesis through depth estimation, damage repair, bidirectional state space modeling and multi-plane image optimization, which solves the problems of structural fracture, occlusion repair distortion and artistic style inaccuracy in multi-perspective synthesis caused by the scattered perspective characteristics of traditional landscape painting. While maintaining the artistic conception of traditional painting, it gives the work an immersive interactive experience, provides technical support for the digital protection of cultural heritage, and significantly improves the visual coherence and detail expression of multi-perspective synthesis.

[0128] The traditional painting multi-view generation method first performs denoising, background unification and edge enhancement pre-processing on the input landscape painting, and then generates a depth map by fusing high and low resolution information into a depth estimation module to accurately depict the virtual and real layers of the picture. By using backtracking and restoration processing combined with edge detection algorithms, the occluded areas caused by perspective transformation are restored to maintain the consistency of the original style. s , D s ) generates discrete viewpoint samples by transforming the virtual camera's pose. It also employs an improved bidirectional state-space model to model long-range dependencies and dynamically optimize semantic associations between depth planes. The resulting output is a continuous 3D scene with dynamic perspectives, exhibiting layered changes consistent with the laws of scattered perspective when the viewpoint switches. This approach preserves the artistic conception of traditional painting while providing an immersive and interactive experience, providing technical support for the digital preservation of cultural heritage.

[0129] Effect Testing: First, Qualitative Comparison: This study aims to evaluate the performance of different 3D Photo-based models in artistic image reconstruction by generating multi-view results for a single landscape painting. We preprocessed the output of each model into a 90-frame video stream format and selected the first frames generated by the MINE, VMPI, AdaMPI, and VistaMPI models for comparative analysis with the true target value (original image). Considering the characteristics of multi-view synthesis, we also selected frames 4 and 10 from the sequence as representative perspectives to ensure the comprehensiveness and fairness of the comparison.

[0130] like Figure 8As shown in the figure, different models show significant differences in generating landscape paintings, and VistaMPI shows better visual effects than other comparison models. MINE fails to fully capture complex texture information during the feature extraction process, resulting in obvious blurring of the generated image, especially unnatural loss of details in the edge area. This problem is mainly attributed to the shortcomings of its depth prediction module in detail modeling. Although VMPI can generate multi-perspective content, the overall color tone is bright and lacks the rustic style and delicate texture of traditional landscape paintings, which affects the artistic expression of the work. In addition, its perspective transformation strategy is relatively conservative, resulting in insufficient changes between different views, reducing the expressiveness of the immersive experience.

[0131] The 3D Photo model suffers from oversharpening edges. While this improves image clarity in some scenes, it also results in a loss of fine textures, undermining the image's overall naturalness. AdaMPI performs poorly when faced with images containing large numbers of irregular holes, failing to effectively restore texture detail in the missing areas and resulting in a loss of image consistency. This suggests that the model is acceptable for scenes with simple structures or minimal textures, but is insufficient for complex structures.

[0132] In comparison, VistaMPI demonstrates a clear advantage. By introducing a filling module and depth prediction mechanism, the model effectively preserves the rich details and hierarchical structure of landscape paintings during the generation process, while also achieving a natural transition between depth relationships, more consistent with human visual perception. This capability is particularly prominent in multi-view synthesis tasks, enabling VistaMPI to outperform other models in terms of visual coherence, realism, and artistry, demonstrating greater robustness and application potential.

[0133] Second, quantitative evaluation: We employ widely recognized image quality metrics, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Perceptual Image Quality Index (LPIPS), to systematically assess the visual quality and realism of generated images. We use the commonly used image quality assessment metrics LPIPS, SSIM, and PSNR to evaluate the quality of images from new perspectives. LPIPS is a learned perceptual image patch similarity metric used to measure the perceptual similarity between images. It is a perceptual similarity metric learned through neural network learning. Lower LPIPS values ​​indicate closer and more similar images. SSIM is the structural similarity index, which measures the degree of similarity between two images. PSNR, the peak signal-to-noise ratio, measures the quality of image reconstruction. Higher PSNR values ​​indicate better image quality and are commonly used to evaluate the performance of compression algorithms. By combining these metrics, we evaluate the generated multi-view images to measure the performance of the model. The PSNR, SSIM, and LPIPS were compared with the current advanced multi-view synthesis methods. As shown in Table 1, the VistaMPI model showed excellent advantages in all evaluation indicators, surpassing existing similar technologies.

[0134] Table 1 Quantitative comparison results with advanced multi-view synthesis methods

[0135] Method PSNR(↑) SSIM(↑) LPIPS(↓) MINE 17.028 0.655 0.639 VMPI 13.638 0.550 0.364 3D Photo 21.389 0.876 0.205 AdaMPI 23.704 0.799 0.183 VistaMPI 34.820 0.936 0.108

[0136] This application uses a widely recognized image quality metric, and the experimental results are shown in Table 1. It can be seen that VistaMPI significantly improves the PSNR indicator by 13.28% over the second-best method AdaMPI, which directly verifies the difference.

[0137] In the SSIM evaluation, VistaMPI achieved a 4.81% improvement in accuracy compared to the classic method 3DPhoto, fully demonstrating the model's ability to accurately reconstruct complex texture structures and spatial hierarchical features.

[0138] It is worth emphasizing that VistaMPI's LPIPS score is as low as 0.108, which achieves a 29.87% improvement in perceptual quality compared to AdaMPI, proving that the multi-view images synthesized by this model are closer to human visual perception.

[0139] The above quantitative results verify VistaMPI's ability to accurately reconstruct complex texture structures and spatial hierarchical features, as well as the high consistency of pixel-level fidelity between the synthesized images and the original images.

[0140] In summary, the experimental results clearly demonstrate the outstanding contribution of the VistaMPI model in the field of multi-view image synthesis, reaching a leading level in both technical accuracy and visual experience.

[0141] In order to verify the effectiveness of the Bi-directional State Space (Bi-SSM) module and the Multi-Head SelfAttention mechanism, this study conducted an ablation experiment on the module. The experimental results are as follows: Figure 6 As shown in Figure 2, it can be seen intuitively that when Bi-SSM and MSA are used together, the generated new perspective is clearer and the structure is more consistent. If the key module is removed, the synthesis result will have problems such as blurred edges or loss of details.

[0142] Table 2 Ablation study of network design

[0143] Method PSNR(↑) SSIM(↑) LPIPS(↓) w / o Bi-SSM 32.160 0.906 0.149 w / o MSA 32.101 0.913 0.137 w / o BiSSM&MSA 31.880 0.898 0.126 VistaMPI 34.820 0.936 0.108

[0144] Experimental results show that VistaMPI significantly improves the ablation model across all test samples. Compared to the ablation results for the improved bidirectional state-space model, the VistaMPI model reduces LPIPS by 37.90% and improves PSNR by 7.63%, demonstrating that the bidirectional state-space module excels in extracting global information, effectively capturing multi-plane disparity distribution and optimizing the overall structural consistency of the image.

[0145] To further evaluate the role of the multi-head self-attention mechanism in generating multi-view images, detailed ablation experiments were conducted. The experimental results are shown in Table 2. Compared to the model with the multi-head self-attention (MSA) ablation, the VistaMPI model achieves a 10.68% improvement in PSNR and a 26.85% reduction in LPIPS. This demonstrates that the multi-head self-attention mechanism has significant advantages in enhancing local feature interactions and refining image details.

[0146] To evaluate the combined effects of the multi-head attention mechanism and the bidirectional state space, ablation experiments were conducted on both modules. The experimental results are shown in Table 2. PSNR and SSIM experienced the most severe degradation, while LIPLS experienced the least degradation. The Bi-SSM and multi-head attention mechanism are complementary, and their combination results in clearer and more consistent structures for the generated new perspective images.

[0147] To verify the effectiveness of the loss function, we set up a set of ablation experiments on the hyperparameters of VistaMPI loss function, including pixel-level error metric loss Perceptual loss Focus frequency loss and smoothing loss

[0148] The experimental results are as follows Figure 7 As shown in , it can be seen intuitively that the ablation experiment results show that there are significant differences in the impact of different loss terms on the generated quality. After the loss, due to the lack of pixel-level constraints, the generated image will have color shift and edge blur. After the loss, the texture structure is abnormal, and the local area appears blurred and distorted compared to the original image on the left. The loss leads to global resolution degradation and color shift.

[0149] Table 3 Ablation study of network design

[0150]

[0151] In order to further quantify the effectiveness of the joint loss function, a series of image quality indicators are used for evaluation, as shown in Table 3. The experimental results show that VistaMPI has a significant improvement in all test samples compared with the ablation model. When , PSNR drops sharply from 34.82 to 26.34 (down 24.3%), indicating that pixel-level reconstruction loss is irreplaceable for basic geometric recovery.

[0152] When removing , LPIPS increased from 0.108 to 0.141 (an increase of 30.6%), verifying its effectiveness in improving visual perception quality through VGG feature constraints.

[0153] When removing When , all indicators are severely degraded, highlighting their key role in suppressing depth discontinuity artifacts.

[0154] Step 4: If Figure 9 As shown in the figure, by inputting the image to be generated and its depth map into the trained VistaMPI model, the multi-view landscape painting image result with a total of 90 frames in 3 seconds can be inferred. Figure 9 This is the generated result of 6 consecutive frames. It can be seen that there are certain perspective differences between different frames. For example, there are certain content differences between the position of the seal in the lower right corner of the picture and the background mountain scenery in different frames. When the consecutive frames are combined into a complete video, the perspective change can be clearly seen.

[0155] Quantitative evaluation: Using metrics such as PSNR, SSIM, and LPIPS, the VistaMPI model surpasses existing similar technologies in various indicators. Ablation experiments also verify the effectiveness of the improved bidirectional state-space model (Bi-SSM) and multi-head self-attention module and joint loss function.

[0156] The present invention proposes a multi-perspective synthesis technology for traditional Chinese landscape paintings, which achieves the goal of generating high-fidelity multi-perspective sequences from a single image through the collaborative optimization of depth estimation, damage repair and bidirectional state space modeling.

[0157] The core of multi-view synthesis lies in the combination of an improved bidirectional state-space model (Bi-SSM) and a multi-plane generation (MPI) framework. First, the image and depth map are downsampled to generate an initial set of depth planes, from which global features are extracted through a five-layer encoder-pooling module. Subsequently, the Bi-SSM recursively models long-range dependencies through forward and backward convolutions, and combines this with a multi-head attention mechanism to focus on key areas (such as mountain contours and cloud transitions) and dynamically optimize cross-plane semantic associations. The ResMamba feature extractor further integrates multi-scale information, simulating the virtual-real hierarchy of traditional painting through a "soft assignment" strategy of foreground / background masks, ultimately outputting the color and density of each plane. Ablation experiments show that removing the Bi-SSM increases LPIPS from 0.108 to 0.149, validating its key role in structural consistency.

[0158] Experimental validation: Compared with methods such as MINE, VMPI, 3D Photo, and AdaMPI, the VistaMPI model outperforms other methods in terms of detail preservation, color, depth transitions, and visual fidelity when generating multi-view images of landscape paintings. Ablation experiments demonstrate that the dual-state space module, multi-head attention mechanism module, and joint loss function play a key role in improving image quality and visual effects. All evaluation metrics confirm the model's advantages.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating multiple views of a traditional image by combining a bidirectional state space model, characterized in that: The following steps are involved: Step 1: Get Traditional Painting I s , for traditional painting images I s Perform low-resolution global estimation and high-resolution local estimation respectively, and fuse the estimation results to generate a depth map D s , get the corresponding image pair (I s , D s ); Step 2: The image pair (I s , D s ) to repair the damage and obtain the new perspective image pair (I t , D t ); Step 3: Input the new perspective into (I t , D t ) as training data to train the traditional painting multi-view generation model; The traditional multi-view generation model of the painting includes at least an improved bidirectional state space model and a multi-plane generation module; The improved two-way state space model uses the two-way state space model to analyze the new perspective input pair (I t , D t ) performs forward and reverse convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations to capture long-range dependencies in scattered perspectives; a multi-plane generation module is used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r Use "soft distribution" strategy to simulate the virtual-real transition of traditional painting, outputting the color and density of each plane; Step 4: Input the test traditional painting image and its corresponding depth map into the trained traditional painting multi-view generation model, and output the multi-view traditional painting image.

2. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 1, characterized in that: The damage repair method in step 2 comprises: s , D s ) The image pair is input into the virtual camera, and the virtual camera pose is transformed Generate intermediate damage view (I m ′,D m ′), using the disparity mask M d Identify the occluded area, combine the ink line structure prior and depth information to repair the damaged area, and obtain the composite edge structure E c ; The dual-branch network L processes color and depth information, and uses PatchGAN as a discriminator to perform local perception of the image and evaluate edge structure and texture details. The expression is: in, Represents the connected composite edge structure E c and the disparity mask M d ⊙ represents the fusion of real edge content and generated edge content to obtain a new perspective input pair (I t , D t ).

3. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 1, characterized in that: In step 3, the traditional multi-view generation model also includes a multi-modal feature encoding module for extracting new perspective input pairs (I t , D t ) Global structural features, establishing physical associations between color and geometry; First, the new perspective input pair (I t , D t ) is downsampled to generate an initial depth plane set Then the new perspective is input to (I t , D t ) and the depth plane set Channel splicing is performed to form multi-channel input features, which are fed into a five-layer encoder-pooling architecture to extract global structural features T 0 , establish the physical association between image color and geometry, the expression is: in represents the s-th layer coding module, represents the s-th layer downsampling module, Represents the cascade operation of five layers of encoding-pooling modules.

4. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 1, characterized in that: Improved two-way state space model, using the two-way state space model to the new perspective input pair (I t , D t ) performs forward and backward convolution recursively to model long-range dependencies, and combines the multi-head attention mechanism to dynamically optimize cross-plane semantic associations and capture long-range dependencies in scattered perspectives, specifically including: For the global structural feature T 0 Perform linear mapping to generate bidirectional feature bases x and z. For the feature base x, perform forward and reverse processing on it to obtain features in two directions, and recursively update the hidden state. in, represents the forward one-dimensional convolution, To flip the sequence x and then perform the convolution operation; The feature base z is activated by the SiLU function to obtain z′, which is combined with the bidirectional features and the global structure and overall content features of the image to obtain the global temporal information T of the image: A multi-head attention mechanism is used for the global temporal information T of the image to focus on key areas and optimize depth calibration.

5. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 1, wherein: Multi-plane generation module, used to extract new perspective input pairs (I t , D t ) and the multi-scale features of the global temporal information T, through the decoder D r A "soft distribution" strategy is used to simulate the virtual-real transition of traditional paintings, outputting the color and density of each plane. Specifically, it includes: The new perspective obtained in step 2 is input to (I t , D t ) and the global temporal information T along the channel dimension, and then input to the ResMamba feature extractor, which includes a feature extraction module and a bidirectional state space model connected in sequence, extracts multi-scale features F, models the dependency between different image regions, and divides them into feature masks M using a multi-plane feature mask. i , background mask and foreground mask Through the decoder D r Adopt "soft allocation" strategy to simulate the virtual-real transition of traditional painting and output the color c of each plane i and density σ i , the expression is:

6. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 1, characterized in that: The method adopts a joint loss function in the process of training the multi-view synthesis model, including: Pixel-level error metric loss, structural similarity loss, perceptual loss, focal frequency loss, and depth regularization loss; The depth regularization loss includes depth sorting loss and depth distribution loss, which are used to ensure the reasonable arrangement of depth values ​​and close to the actual depth distribution; The pixel-level error metric loss is used to measure the pixel-level error between the generated image and the real image, constraining the pixel-level accuracy so that the generated image retains the artistic style and details of the original as much as possible. The expression is: in, Indicates the generated image, Represents the image of the input pair, H and W represent the height and width of the image respectively; i, j is a pixel at a specific position in the image; The perceptual loss extracts high-level semantic features through a pre-trained convolutional neural network and compares the differences between the generated image and the target image in the feature space, thereby capturing the brushstrokes, texture, and style information in the painting. The expression is: Among them, φ l (·) represents the feature extractor output of the l-th layer of the convolutional neural network, N l represents the total number of elements in the feature map of the lth layer, and L represents the number of selected feature layers; Structural similarity loss enhances the perceptual quality of the image by evaluating the consistency of brightness, contrast, and structural information between the generated image and the target image in a local area. The expression is: Among them, μ (·) represents the local mean of the image, represents the local covariance between the two, represents the local variance of the image, C (·) is a small constant used to avoid zero denominator and improve numerical stability; Focusing on frequency loss, by analyzing the frequency domain distribution of the image, we can enhance the recovery of high-frequency components while protecting the continuity of low-frequency components. The expression is: Where F(x) represents the Fourier transform of the image, α>0 and β>0 are hyperparameters that control the degree of attention to different frequencies, and i represents the index in the frequency domain; At the same time, the depth regularization loss, Used to constrain the physical rationality of the relationship between MPI layers and network output, ensure that the prediction value of the traditional multi-view generation model meets the actual requirements, and add regularization loss The expression is as follows: The depth sorting loss Used to ensure that the depth values ​​are arranged in a reasonable order to avoid unreasonable jumps or intersections; depth allocation loss Used to make the predicted depth value closer to the actual depth distribution D s , thereby improving geometric consistency, the expression is: Among them, d i Indicates the depth of the i-th layer.

7. The method for generating multiple views of a traditional image by combining a bidirectional state space model according to claim 6, characterized in that: The joint loss function also includes a smoothing loss, which is used to ensure the depth continuity of the distant part and prevent the depth prediction from jumping sharply; Smoothing loss The expression is: in, represents the gradient of the depth map, represents the gradient of the color image, Used to dynamically adjust the smoothing strength based on the color gradient.

Images