Rapid three-dimensional style migration method for sparse view of double-branch feed-forward network structure

Abstract

The invention discloses a sparse view rapid three-dimensional style migration method of a double-branch feed-forward network structure. The method comprises the following steps: S1, extracting three-dimensional structure information; s2, fusing style features and predicting colors; s3, synthesizing a stylized three-dimensional Gaussian scene; and S4, fine tuning the stylized three-dimensional Gaussian scene: keeping the geometric structure of the stylized three-dimensional Gaussian scene unchanged, and performing fine tuning and optimization on the appearance. According to the method, rapid stylized three-dimensional modeling can be realized by using a small number of input images, compared with the existing method, the rendering time is remarkably shortened, the problems of slow rendering speed and incapability of real-time interaction of the old method and the problems of geometric structure distortion (structure distortion) and inconsistent visual angles caused by style migration are solved, and the method has the advantages of being high in practicability and easy to popularize. The combined learning ability of new view synthesis and three-dimensional stylization is enhanced; real-time application can be supported, and the quality is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and three-dimensional reconstruction, and in particular to a sparse view fast three-dimensional style transfer method with a dual-branch feedforward network structure. Background Art

[0002] In computer graphics and computer vision, reconstructing 2D images into 3D scenes, especially 3D reconstruction with a specific artistic style, has always been an important and challenging task. Traditional 3D style transfer methods [1], [2], [3] mostly rely on models such as neural radiance fields [4] or 3D Gaussian splatter [5], and require a large number of input viewpoints with pose information [6] and complex test-time optimization processes [7]. These methods are not only inefficient but also unsuitable for general users.

[0003] The main problems with existing technologies are: First, traditional methods rely on large numbers of multi-view images with pose annotations, often requiring specialized acquisition equipment or additional calibration, which increases the barrier to use. Second, existing 3D style transfer methods typically employ a test-time optimization strategy, performing a separate optimization for each scene or style. This process often takes minutes to hours and is difficult to meet the requirements of interactive applications. Third, injecting style features into the 3D representation can easily lead to structural distortion or perspective inconsistency, which can easily disrupt the consistency of the 3D structure during the style transfer process, affecting the usability of the 3D scene. These problems urgently need to be addressed.

[0004] The difficulty in addressing these issues and drawbacks lies in the following: Achieving fast stylized 3D modeling with sparse, pose-free image input requires addressing multiple challenges simultaneously. First, a sound network architecture must be designed to decouple shape modeling from appearance modeling, addressing the information loss and geometric distortion associated with direct style transfer from images. Second, the model must generate stylized 3D scenes in a short period of time, requiring fast network inference and no test-time optimization.

[0005] The significance of solving the above problems and defects is: it provides a practical idea for quickly generating stylized three-dimensional scenes using a small number of images, without the need for complex shooting processes and camera parameter input, and can be applied to real-time interactive applications. It solves many problems of the old method and is of great help to the rapid production of digital content. Summary of the Invention

[0006] The present invention provides a method for fast 3D style transfer of sparse views with a dual-branch feedforward network structure, which can achieve fast stylized 3D modeling using a small number of input images. Compared with existing methods, the present invention significantly reduces rendering time, solves the problems of slow rendering speed and inability to interact in real time of the old methods, as well as the problems of geometric structure distortion (structural distortion) and perspective inconsistency caused by style transfer, enhances the joint learning capability of new view synthesis and 3D stylization, supports real-time applications, and improves quality.

[0007] The technical solutions of the present invention are as follows: The present invention discloses a method for fast 3D style transfer from sparse views using a dual-branch feedforward network structure, comprising the following steps: S1. Extracting 3D structural information: In the structure branch, a structural encoder that shares information with other viewpoints is used to extract features from each input image, and then a structural decoder is used to fuse multi-view features to predict the structural properties of the 3D Gaussian; S2. Fusion of style features and color prediction: In the appearance branch, a style image encoder is used to extract style features, and the style and structural features are fused through a cross-style-structure cross-attention module to obtain a stylized appearance representation, and then the color information of each Gaussian point is predicted; S3. Synthesizing a stylized 3D Gaussian scene: The outputs of the structure branch and the appearance branch are integrated into a complete stylized 3D Gaussian scene; S4. Fine-tuning the stylized 3D Gaussian scene: The geometric structure of the stylized 3D Gaussian scene is kept unchanged, and the appearance is fine-tuned and optimized.

[0008] Optionally, in the sparse view fast 3D style transfer method of the dual-branch feedforward network structure, in step S1, first, a set of sparse input images without camera pose is given. , where each image has a size of ,in and The height and width of the image are respectively, and the structural parameters of the three-dimensional Gaussian are predicted through the structure branch; the input image set is converted into Encoded as a collection of structural features , whose expression is:

[0009] in is the block size of the image.

[0010] Subsequently, the structural features of each view are decoded through the structural decoder to perform cross-view attention structure decoding, while sharing information with other views, which is expressed as:

[0011] Two dense prediction heads (DPTs) are used to predict the structural parameters of the three-dimensional Gaussian: one is used to output the three-dimensional Gaussian center position parameters , and the other outputs the remaining parameters, including rotation , Zoom and opacity ; Get the set of three-dimensional Gaussian structure parameters .

[0012] Optionally, in the above-mentioned sparse view fast 3D style transfer method with a dual-branch feedforward network structure, in step S2, for the input single style image , which is encoded into style features through the style encoder , expressed as:

[0013] Perform the following two processes: First, the set of structural features from all perspectives Through the global self-attention module Get the set of enhanced structural features :

[0014] Then use the style-structure cross-attention module The style features and the enhanced set of structural features Perform feature fusion to obtain fusion features , expressed as:

[0015] in 、 、 is a learnable projection matrix; Similar to step S1, a dense prediction head is used as a style decoder to predict the color parameters of each 3D Gaussian , get the set of three-dimensional Gaussian color parameters , expressed as: .

[0016] Optionally, in the above-mentioned sparse view fast 3D style transfer method of the dual-branch feedforward network structure, in step S3, the set of 3D Gaussian structure parameters outputted in step S1 is converted into The set of three-dimensional Gaussian color parameters output from step S2 Merge to get a set of stylized 3D Gaussians , expressed as: .

[0017] Optionally, in the above-mentioned sparse view fast 3D style transfer method with a dual-branch feedforward network structure, in step S4, a two-stage training strategy is designed: a pre-training stage and a style transfer fine-tuning stage, wherein: In the pre-training phase, the input image set At the same time, it is used as the input image and style image, and enters the structure and appearance branches through the dual-branch feedforward network, so that the color output is the original color parameter , forming a collection of non-stylized 3D Gaussian scenes , expressed as:

[0018] In the pre-training stage, the renderer generates new perspective images and compares them with the actual input images to optimize the photometric loss. , the loss includes and , expressed as: ; In the style transfer fine-tuning stage, let the set of three-dimensional Gaussian structure parameters Keep unchanged and input image set and style images As input, only the appearance branch is optimized; a loss function is designed, expressed as: .

[0019] in Computational rendering With style images Rectified Linear Units in Very Deep Convolutional Networks (VGG) arrive Differences in feature mean and variance across layers; Using rectified linear units and Layers maintain structural information; is the weighted coefficient of style loss; After optimizing the appearance branch, a stylized 3D Gaussian scene is obtained. .

[0020] According to the technical solution of the present invention, the beneficial effects produced are: The sparse view fast 3D style transfer method of the dual-branch feedforward network structure of the present invention benefits from the explicit characteristics of the 3D Gaussian and the generation ability of the diffusion model. A dual-branch feedforward neural network structure for 3D stylization is designed, which does not require a test-time optimization strategy, significantly reduces the rendering time of the 3D style transfer, maintains the consistency of the 3D structure, and ensures the stability of the generated 3D scene structure. The 3D structure modeling and appearance style transfer are decoupled to achieve rapid completion of 3D stylized modeling, solving the problems of slow rendering speed, inability to interact in real time, real-time structural distortion and inconsistent perspective of the old method. Specifically, the present invention designs a dual-branch network structure that decouples appearance and structure modeling, which is responsible for predicting the color information and geometric properties of the 3D Gaussian respectively, enhancing the joint learning ability of new view synthesis and 3D stylization, and solving the problems of structural distortion and inconsistent perspective of the old method.

[0021] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below with reference to the accompanying drawings and through specific embodiments: BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0023] Figure 1 Flowchart of the sparse view fast 3D style transfer method of the dual-branch feedforward network structure of the present invention; Figure 2 It is a flow chart of the dual-branch feedforward network structure; Figure 3 This is a comparison chart of the results of fast 3D style transfer of sparse views between the present invention and the existing method. DETAILED DESCRIPTION

[0024] To make the purpose, technical methods and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the present invention.

[0025] Based on a feedforward neural network and three-dimensional Gaussian splattering, the present invention realizes the rapid output of three-dimensional style transfer from sparse views and images, providing a fast and high-quality three-dimensional style transfer method. Specifically, a dual-branch feedforward network structure is designed to decouple structure and appearance, extract a consistent spatial representation without the need for camera parameter input, and fuse the features of the style image with the structural features under multiple perspectives. A two-stage training strategy is adopted during training to achieve three-dimensional style transfer while ensuring the consistency of the three-dimensional structure.

[0026] The principles of the present invention are as follows: 1) a dual-branch feedforward network is used to decouple the three-dimensional structure from the appearance, and the geometric properties and color information of the three-dimensional Gaussian are predicted separately; 2) on the structure branch, unsupervised multi-view feature fusion is used to construct the three-dimensional structure, and a consistent spatial representation is extracted without the need for camera parameter input; 3) on the appearance branch, a style-structure cross-attention mechanism is introduced to fuse the features of the style image with the structural features under multiple viewpoints to achieve a consistent expression of the appearance style; 4) a two-stage training strategy is adopted: in the first stage, new viewpoint synthesis training is performed to obtain a structurally faithful Gaussian representation, and in the second stage, the appearance branch is fine-tuned on the basis of the frozen structure branch to inject style features.

[0027] like Figure 1 As shown, the sparse view fast 3D style transfer method of the dual-branch feedforward network structure of the present invention includes the following steps from start to finish: S1. Extracting 3D structural information: In the structure branch, a structure encoder that shares information with other viewpoints is used to extract features from each input image. The multi-view features are then fused through a structure decoder to predict the structural properties of the 3D Gaussian.

[0028] In this step, the structural parameters of the three-dimensional Gaussian are first predicted through the structural branch; according to the input image set Get the set of three-dimensional Gaussian structure parameters .

[0029] S2. Fusion of style features and color prediction: In the appearance branch, a style image encoder is used to extract style features. The style and structure features are fused through a cross-style-structure cross-attention module to obtain a stylized appearance representation, and then the color information of each Gaussian point is predicted.

[0030] In this step, the style encoder extracts the style features and the style-structure cross-attention module Fusion, predicting the color parameters of each Gaussian Get a set of three-dimensional Gaussian color parameters .

[0031] S3. Synthesizing Stylized 3D Gaussian Scenes: Integrate the outputs of the structure branch and the appearance branch into a complete stylized 3D Gaussian scene.

[0032] In this step, the set of three-dimensional Gaussian structure parameters A collection of appearance parameters Merge to get a set of stylized 3D Gaussians .

[0033] S4. Fine-tuning the Stylized 3D Gaussian Scene: The geometric structure of the stylized 3D Gaussian scene remains unchanged, while fine-tuning and optimizing its appearance.

[0034] In this step, a two-stage training strategy and loss function are designed. In the fine-tuning stage, the set of three-dimensional Gaussian structure parameters is set to Keep it unchanged, fine-tune the color parameters, and get the optimized stylized 3D Gaussian scene .

[0035] The method of the present invention decouples the three-dimensional structure from the appearance through a dual-branch feedforward network, and predicts the geometric properties and color information of the three-dimensional Gaussian separately: on the structure branch, unsupervised multi-view feature fusion is used to construct the three-dimensional structure, and a consistent spatial representation is extracted without the need for camera parameter input; on the appearance branch, a style-structure cross-attention mechanism is introduced to fuse the features of the style image with the structural features under multiple perspectives to achieve a consistent expression of the appearance style. In terms of training strategy, the present invention uses a two-stage training strategy, focusing on generating accurate position parameters in the pre-training stage, and retaining the original reconstruction capability of the model and performing fine-tuning in the fine-tuning stage to prevent structural degradation caused by style transfer. The specific implementation steps are as follows: S1. Extracting 3D structural information: First, given a set of sparse input images without camera poses, , where each image has a size of ,in and The height and width of the image are respectively. First, the structural parameters of the three-dimensional Gaussian are predicted through the structural branch; the input image set is converted into Encoded as a collection of structural features , whose expression is:

[0036] in is the block size of the image.

[0037] Subsequently, the structural features of each view are decoded through the structural decoder to perform cross-view attention structure decoding, while sharing information with other views, which is expressed as:

[0038] Two dense prediction heads (DPTs) are used to predict the structural parameters of the three-dimensional Gaussian: one is used to output the three-dimensional Gaussian center position parameters , and the other outputs the remaining parameters, including rotation , Zoom and opacity ; This results in a set of three-dimensional Gaussian structure parameters: .

[0039] S2. Fusion style features and color prediction: For a single input style image , which is encoded into style features through the style encoder , expressed as:

[0040] Perform the following two processes: First, the set of structural features from all perspectives Through the global self-attention module Get the set of enhanced structural features :

[0041] Then use the style-structure cross-attention module The style features and the enhanced set of structural features Perform feature fusion to obtain fusion features , expressed as:

[0042] in 、 、 is the learnable projection matrix.

[0043] Similar to step S1, a dense prediction head is used as a style decoder to predict the color parameters of each 3D Gaussian , get the set of three-dimensional Gaussian color parameters , expressed as: .

[0044] S3. Synthesize stylized 3D scenes: The set of three-dimensional Gaussian structure parameters output from step S1 The set of three-dimensional Gaussian color parameters output from step S2 Merge to get a set of stylized 3D Gaussians , expressed as:

[0045] The flow chart of the dual-branch feedforward network structure is as follows Figure 2 shown.

[0046] S4. Fine-tune the stylized 3D scene: The present invention designs a two-stage training strategy.

[0047] In the first stage, the pre-training stage, the input image set At the same time, it is used as the input image and style image, and enters the structure and appearance branches through the dual-branch feedforward network, so that the color output is the original color parameter , forming a collection of non-stylized 3D Gaussian scenes , expressed as:

[0048] Specifically, in the pre-training stage, the renderer generates new perspective images and compares them with the actual input images to optimize the photometric loss. , the loss includes and , expressed as:

[0049] In the second stage, that is, the style transfer fine-tuning stage, let the set of three-dimensional Gaussian structure parameters Keep unchanged and input image set and style images As input, only the appearance branch is optimized. The present invention designs a loss function, which is expressed as: .

[0050] in Computational rendering With style images Rectified Linear Units in Very Deep Convolutional Networks (VGG) arrive Differences in feature mean and variance across layers; Using rectified linear units and Layers maintain structural information; is the weighting coefficient of style loss.

[0051] After optimizing the appearance branch, a stylized 3D Gaussian scene is obtained. .

[0052] The intuitive effect comparison between the present invention and the existing method can be seen Figure 3 Using a small number of input images, the present invention can better capture the details of the style image while preserving the original scene structure and transfer the style to the 3D scene. In comparison, existing methods are prone to offset in the 3D structure and cannot effectively transfer the style of the style image to the 3D scene.

[0053] Table 1 below compares the results of the present invention and existing 3D style transfer methods, comparing continuity within small and large scales, as well as style transfer time. To facilitate quantitative calculation and comparison, we used learned-perceptual image block similarity and root mean square error (RMSE) as metrics to compare the continuity of the present invention and existing methods across different scales. Lower learned-perceptual image block similarity and RMSSE indicate better model performance, indicating greater consistency in the 3D structure.

[0054] Table 1: Comparison of parameters involved in image generation of dynamic 3D scenes between the present invention and existing models

[0055] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the scope of protection of the present claims. Other embodiments and combinations of implementations of the present invention are also within the scope of protection of the present invention.

[0056] References [1]K. Zhang et al. , “ARF: Artistic Radiance Fields,” in Computer Vision – ECCV 2022 , vol. 13691, S. Avidan, G. Brostow, M. Cissé, GM Farinella, and T. Hassner, Eds., in Lecture Notes in Computer Science, vol. 13691. , Cham: Springer Nature Switzerland, 2022, pp. 717–733. doi: 10.1007 / 978-3-031-19821-2_41. [2]K. Liu, F. Zhan, M. Xu, C. Theobalt, L. Shao, and S. Lu, “StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting,” in SIGGRAPH Asia 2024 Technical Communications , Tokyo Japan: ACM, Dec. 2024, pp. 1–4. doi: 10.1145 / 3681758.3698002. [3]K. Liu et al., “StyleRF: Zero-shot 3D Style Transfer of NeuralRadiance Fields,” 2023, arXiv . doi: 10.48550 / ARXIV.2303.10598. [4] B. Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM , vol. 65, no. 1, pp. 99–106, 2021. [5] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3dgaussian splatting for real-time radiance field rendering.,” ACM Trans Graph , vol. 42, no. 4, pp. 139–1, 2023. [6] Song Xuan, Zheng Shaoming, Chen Yonghao, Shu Jiayang, Hu Weizhuang, Zhuo Hang, and Chen Ruiyao, “A method and terminal for stylized transfer of 3D images,” CN115588070B [Online]. Available: https: / / patents.google.com / patent / CN115588070B / zh [7] Zhang Ya, Zhang Yexun, Cai Wenbin, and Wang Yanfeng, “Method and system for transferring arbitrary style and content based on a small number of samples,” CN107767328A [Online]. Available: https: / / patents.google.com / patent / CN107767328A / zh.

Claims

1. A fast 3D style transfer method for sparse views with a dual-branch feedforward network structure, characterized by: The following steps are involved: S1. Extracting 3D structural information: In the structure branch, a structure encoder that shares information with other views is used to extract features from each input image. The structure decoder then fuses these multi-view features to predict the structural properties of the 3D Gaussian. S2. Fusion of style features and color prediction: In the appearance branch, a style image encoder is used to extract style features. Style and structural features are then fused via a cross-style-structure cross-attention module to obtain a stylized appearance representation. The color information of each Gaussian point is then predicted. S3. Synthesizing a stylized 3D Gaussian scene: Integrating the outputs of the structure branch and the appearance branch into a complete stylized 3D Gaussian scene; S4. Fine-tuning the stylized 3D Gaussian scene: The geometric structure of the stylized 3D Gaussian scene remains unchanged, while fine-tuning and optimizing its appearance.

2. The sparse view fast 3D style transfer method with a dual-branch feedforward network structure according to claim 1 is characterized in that: In step S1, first, given a set of sparse input images without camera poses , where each image has a size of ,in and are the height and width of the image respectively. First, the structural parameters of the three-dimensional Gaussian are predicted through the structural branch; the input image set is converted into Encoded as a collection of structural features , whose expression is: in is the block size of the image; Subsequently, the structural features of each view are decoded through the structural decoder to perform cross-view attention structure decoding, while sharing information with other views, which is expressed as: Two dense prediction heads (DPTs) are used to predict the structural parameters of the three-dimensional Gaussian: one is used to output the three-dimensional Gaussian center position parameters , and the other outputs the remaining parameters, including rotation , Zoom and opacity ; Get the set of three-dimensional Gaussian structure parameters .

3. The method for fast 3D style transfer from sparse views using a dual-branch feedforward network structure according to claim 2, characterized in that: In step S2, for the input single style image , which is encoded into style features through the style encoder , expressed as: Perform the following two processes: First, the set of structural features from all perspectives Through the global self-attention module Get the set of enhanced structural features : Then use the style-structure cross-attention module The style features With the enhanced structural features Perform feature fusion to obtain fusion features , expressed as: in 、 、 is a learnable projection matrix; Similar to step S1, a dense prediction head is used as a style decoder to predict the color parameters of each 3D Gaussian , get the set of three-dimensional Gaussian color parameters , expressed as: 。 4. The method for fast 3D style transfer from sparse views using a dual-branch feedforward network structure according to claim 2, characterized in that: In step S3, the set of three-dimensional Gaussian structure parameters output in step S1 is The set of three-dimensional Gaussian color parameters output from step S2 Merge to get a set of stylized 3D Gaussians , expressed as: 。 5. The method for fast 3D style transfer from sparse views using a dual-branch feedforward network structure according to claim 2, characterized in that: In step S4, a two-stage training strategy is designed: a pre-training stage and a style transfer fine-tuning stage, where In the pre-training phase, the input image set At the same time, it is used as the input image and style image, and enters the structure and appearance branches through the dual-branch feedforward network, so that the color output is the original color parameter , forming a collection of non-stylized 3D Gaussian scenes , expressed as: In the pre-training phase, a new perspective image is generated by the renderer and compared with the actual input image to optimize the photometric loss. , the loss includes and , expressed as: ； In the style transfer fine-tuning stage, let the set of three-dimensional Gaussian structure parameters Keep unchanged and input image set and style images As input, only the appearance branch is optimized; a loss function is designed, expressed as: ； in Computational rendering With style images Rectified Linear Units in Very Deep Convolutional Networks (VGG) arrive Differences in feature mean and variance across layers; Using rectified linear units and Layers maintain structural information; is the weighted coefficient of style loss; After optimizing the appearance branch, a stylized 3D Gaussian scene is obtained. .

Images