A 3D ink-wash style transfer method based on single-view multi-plane image guidance

Abstract

The application discloses a 3D ink painting style migration method based on single-view multi-plane image guidance, which comprises the following steps: step 1: constructing multi-plane images MPIs, 1-1: generating single-view multi-plane images MPIs from a single image by using an Encoder-Decoder structure of a full convolutional network; 1-2: depth layer discretization; 1-3: inference process, during the inference process, the Encoder is run only once to extract features; step 2: extracting ink painting style features, defining I s is a content image with a depth of z; 2-1: feature extraction, 2-2: nearest neighbor feature matching NNFM loss, step 3, view synthesis. zi is a content image with a depth of z; 2-1: feature extraction, 2-2: nearest neighbor feature matching NNFM loss, step 3, view synthesis. The application can better utilize single-view multi-plane images (MPIs) to express the layered structure of ink paintings, greatly improve the edge stroke effect and clean white space effect, and at the same time, by adopting the 3D ink painting style migration technical route guided by single-view multi-plane images, the ink painting style migration effect is greatly improved, and new view synthesis of ink painting style scenes is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image processing, and in particular to a 3D ink style transfer method based on single-view multi-plane image guidance. BACKGROUND

[0002] In the field of graphics and computer vision, image style transfer is a challenging task that aims to transform the visual style of one image to another while preserving the semantic content of the original image. This technology has been widely applied in many fields, including artistic creation, film special effects, game design, etc.

[0003] Traditional image style transfer methods usually rely on optimization algorithms, such as the method based on neural style transfer proposed by Gatys et al. These methods achieve image style transformation by minimizing the content loss and style loss between the original image and the target style image. However, these methods are often inefficient in handling large-scale images and complex scenes, requiring a large amount of computational resources and time.

[0004] With the development of deep learning technology, image style transfer methods based on deep neural networks have gradually become mainstream. Deep learning models such as generative adversarial networks (GAN) and convolutional neural networks (CNN) are widely used in image style transfer tasks, achieving better results. These methods usually include a generator network and a discriminator network, which are trained adversarially to achieve image style transformation.

[0005] However, most of the current image style transfer methods are for 2D images, and there are still challenges in the stylization rendering of 3D scenes. In particular, when applying the traditional ink painting style to 3D scenes, due to the unique characteristics of ink painting such as brush strokes, white space, and ink blot effects, existing 3D style transfer methods often fail to effectively capture and express the unique characteristics of ink painting. SUMMARY

[0006] The purpose of the present application is to combine deep learning and traditional image processing technology to transfer the ink painting style to single-view multi-plane images (MPIs), which contain multiple planes Each plane represents the content of the scene at a certain depth. The goal is to better express the hierarchical structure characteristics of ink painting, constrain the edges of objects at different spatial levels, and improve the edge brush effect and clean white space effect. Secondly, NNFM is used to utilize effective features to improve the style transfer effect of the model. Finally, the three-dimensional representation of the camera view frustum is predicted, and the differentiable rendering technology is used to render new perspective images with ink style.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] A 3D ink style transfer method based on single view multi-plane image guidance, comprising the following steps:

[0009] Step 1: Constructing multi-plane image MPIs;

[0010] 1-1: Using the Encoder-Decoder structure of the full convolutional network to generate single view multi-plane image MPIs from a single image, the Encoder uses the Resnet-50 model, the input is the image , the output is the feature representation , the Decoder is designed similar to Monodepth2, the input is the feature output by the Encoder and the disparity value , the output is a set of images at different depths , which contains RGB values and volume density ;

[0011] 1-2: Depth layer discretization;

[0012] 1-3: Inference process, during inference, the Encoder is only run once to extract features, and the Decoder is run N times to generate a set of images , each corresponding to a depth z, generating the entire MPIs;

[0013] Step 2: Extracting ink painting style features, define as the ink painting image that provides the style, is the content image at depth z;

[0014] 2-1: Feature extraction, run the VGG-19 network, extract ink style features from the ink painting image , run the VGG-19 network N times to extract N content features from the content image , get the content feature set , let represent the feature at pixel position ;

[0015] 2-2: Nearest neighbor feature matching NNFM loss: use NNFM loss to transfer the style brush details of the ink painting into the content image, get the content image with style features ;

[0016] Step 3, view synthesis;

[0017] 3-1: Predicting the three-dimensional representation of the camera frustum, defining the content image with style features , using camera intrinsics Converting 2D pixel coordinates to the coordinate representation of 3D points: ;

[0018] 3-2: Differentiable rendering, rendering the reconstructed camera frustum to a new view image using differentiable rendering techniques;

[0019] 3-3: Rendering new view images, rendering a new view image of a camera rotation and translation .

[0020] Preferably, in step 1, the multi-plane image MPIs is a three-dimensional representation consisting of N planes parallel to the reference camera, these planes are located within the camera frustum, and are arranged at equal intervals in depth Z and disparity d.

[0021] Preferably, each of the images is a 4-channel image for representing the content of the plane with depth Z, containing RGB values and volume density ;

[0022] wherein, the transpose symbol T represents a column vector, refers to a three-dimensional real space. A three-dimensional vector is usually represented as [x, y, z], represents the set of positive real numbers, that is, all real numbers greater than zero.

[0023] Preferably, in step 1-2, the specific process of depth layer discretization is: in the frustum of the source camera, define the depth range of the frustum reconstruction , wherein is the near plane, is the far plane, these values are predefined hyperparameters for determining the depth perception range of the scene, and the continuous depth range is discretized into N equally spaced intervals, each interval defines a depth layer, and each depth layer has a boundary value for representing the disparity value in different depth ranges, the boundary value of each interval is calculated by the following formula:

[0024]

[0025] wherein, is the minimum disparity value, is the maximum disparity value, and n is the index of the current depth layer.

[0026] Preferably, in step 2-2: the style brushstroke details of the ink painting are matched from the ink style features using the nearest neighbor feature matching (NNFM) loss are migrated to N depth level content features respectively , which can be written as:

[0027]

[0028] where M is the number of pixels in , and represents the feature at pixel position , and represents the feature at pixel position . For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space, define the cosine distance as the cosine value of the angle between vectors and , which is used to measure the similarity between style features and content features :

[0029]

[0030] For each feature of , we minimize its cosine distance to its nearest neighbor in the VGG-19 feature space of the style image to perform feature matching.

[0031] Preferably, step 3-1: the conversion from perspective three-dimensional coordinates to Cartesian coordinates is :

[0032]

[0033] where , is a constant representing the distance between the layer and the layer:

[0034] The mapping from two-dimensional real space to the positive real set is:

[0035]

[0036] Preferably, step 3-2 can be represented as:

[0037]

[0038] wherein, is the rendered image, is the number of depth levels, is the cumulative transparency from the first layer to the jth layer, is the cumulative transparency from the first layer to the jth layer, and denote the volume density value and RGB value of the jth layer MPI respectively, denotes the two-dimensional real space, denotes the positive real set, i.e. all real numbers greater than zero.

[0039] Preferably, step 3-3 of rendering the RGB image under any new view needs additional steps:

[0040]

[0041] wherein is the normal vector of the plane , the operation is defined as , and the projection of the target camera is calculated wherein ,

[0042] denotes the Euclidean distance between the two intersection points:

[0043]

[0044] After this step, replace , , with , , respectively, and the new view image can be rendered by the differentiable rendering formula.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] The present application can better utilize single-view multi-plane images (MPIs) to express the layered structure of ink painting, greatly improving the effect of edge strokes and clean white space effect. At the same time, by adopting the single-view multi-plane image guided 3D ink style transfer technology route, the ink style transfer effect is greatly improved, and the new view synthesis of ink style scenes is realized.

[0047] The present application predicts the three-dimensional expression of the camera view cone by the single-view multi-plane image guided method, and uses this three-dimensional expression to give the relative position and angle change of the target camera in three-dimensional space relative to the source camera, thereby achieving the purpose of efficiently rendering the RGB image under the target camera view. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The technical roadmap of 3D ink style transfer guided by single-view multi-plane images in this invention;

[0049] Figure 2 The present invention uses the nearest neighbor feature matching NNFM style stroke detail migration map.

[0050] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] The background of the proposal proposed in this application is:

[0053] With the widespread application of visual content creation in mobile photography and augmented reality / virtual reality (AR / VR), traditional image forms have become unable to meet the growing practical needs. In particular, with the increasing demand for creating 3D content from 2D image assets, there is an urgent need to study a method that can expand traditional ink painting to dimensions beyond the 2D image plane.

[0054] In the 3D stylization problem, ensuring that the stylized result is perceptually similar to the style of a given reference image is a key challenge. Existing 3D scene style transfer methods are mostly based on Western painting styles. However, ink paintings have significant differences in visual features, semantic characteristics, and aesthetic criteria from photographs and oil paintings. Existing 3D style transfer methods usually transfer the overall style of the entire scene, which is contrary to the use of techniques such as outline white space in ink painting style to reflect the characteristics of objects at different levels. As a result, these methods cannot achieve a clean white space effect and have difficulty accurately capturing the unique brushstroke characteristics of ink painting. Therefore, further exploration of 3D style transfer methods based on the characteristics of Chinese ink painting is needed to better preserve and express the artistic characteristics of ink painting.

[0055] Example

[0056] like Figure 1 and Figure 2 As shown, the present application provides a 3D ink style transfer method based on single-view multi-plane image guidance, comprising the following steps:

[0057] Step 1: Construct multi-plane images MPIs. Multi-plane images MPIs are a three-dimensional representation consisting of N planes parallel to the reference camera. These planes are located in the camera frustum and are arranged at equal intervals in depth Z and disparity d. Image is a 4-channel image, which represents the planar content of depth Z, containing RGB values and volume density ;

[0058] wherein, T denotes the column vector, R3 refers to three-dimensional real space, and a three-dimensional vector is usually represented as [x, y, z], R+ refers to the set of positive real numbers, that is, all real numbers greater than zero.

[0059] 1-1: Generate single-view multi-plane images MPIs from a single image using the Encoder-Decoder structure of the full convolutional network, the Encoder uses the Resnet-50 model, the input is the image , the output is the feature representation , the Decoder is designed similar to Monodepth2, the input is the feature output by the Encoder and the disparity value , the output is different depth images, containing RGB values and volume density ;

[0060] 1-2: Depth layer discretization, the specific process of depth layer discretization is: in the view cone of the source camera, define the depth range of the view cone reconstruction , wherein is the near plane, is the far plane, these values are predefined hyperparameters used to determine the depth perception range of the scene, and the continuous depth range is discretized into N equally spaced intervals, each interval defines a depth layer, and each depth layer has a boundary value used to represent the disparity value in different depth ranges, the boundary value of each interval is calculated by the following formula:

[0061]

[0062] wherein, is the minimum disparity value, is the maximum disparity value, and n is the index of the current depth layer.

[0063] 1-3: Inference process, during inference, the Encoder is only run once to extract features, and the Decoder is run N times to generate a set of images , each corresponding to a depth z, generating the entire MPIs;

[0064] Step 2: Extract the style features of ink painting and define To provide a stylish ink painting image, is a content image with depth z, using the VGG-19 network pre-trained on the ImageNet dataset. The network consists of 5 layer blocks: conv1, conv2, conv3, conv4, and conv5. This application uses the conv3 block as the feature extractor.

[0065] 2-1: Feature extraction, running VGG-19 network, from ink painting images Extracting ink painting style features , run the VGG-19 network N times, from the content image Extract N content features , get the content feature set ,make Representation characteristics At pixel position The eigenvector at ;

[0066] 2-2: Nearest Neighbor Feature Matching (NNFM) Loss: Use NNFM loss to transfer the style stroke details of the ink painting to the content image to obtain a content image with style features. ; Use the nearest neighbor feature matching NNFM loss to separate the style stroke details of the ink painting from the ink style features Migrate to N depth-level content features respectively , by minimizing the NNFM loss, the style transfer effect of each MPI plane is adjusted to effectively capture and express the unique characteristics of ink painting, which can be written as:

[0067]

[0068] where M is The number of pixels in Representation characteristics At pixel position The eigenvector at Representation characteristics At pixel position For the eigenvector at For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space, defining the cosine distance is a vector and The cosine of the angle between is used to measure style features. and content features Similarity between:

[0069]

[0070] For each feature of , we minimize its cosine distance to its nearest neighbor in the VGG-19 feature space of the style image .

[0071] Step 3-1: Conversion from perspective three-dimensional coordinates to Cartesian coordinates :

[0072]

[0073] where, , is a constant representing the distance between the layer and the layer:

[0074] The mapping from two-dimensional real space to the positive real set is:

[0075]

[0076] Step 3, view synthesis;

[0077] 3-1: Predict the three-dimensional representation of the camera frustum, define the content image with style features , use the camera intrinsic to convert 2D pixel coordinates to coordinate representation of 3D points: ;

[0078] 3-2: Differentiable rendering, use differentiable rendering technology to render the reconstructed camera frustum to the image of the new view, which can be represented as:

[0079]

[0080] where, is the rendered image, is the number of depth levels, is the cumulative transparency from the first layer to the layer, and represent the volume density value and RGB value of the first layer MPI respectively, refers to two-dimensional real space, represents the positive real set, that is, all real numbers greater than zero.

[0081] 3-3: Render the new perspective image, render a camera rotation and translation ​rendering an RGB image at an arbitrary new viewpoint requires additional steps:

[0082]

[0083] where is the normal vector of the plane This operation is defined as and the projection of the target camera is then computed as where , ,

[0084] denotes the Euclidean distance between two intersection points:

[0085]

[0086] After this step, replace , , with , , respectively, and the new viewpoint image can be rendered by the differentiable rendering formula.

[0087] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that processes, methods, articles, or apparatuses that comprise a list of elements are not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0088] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A 3D ink style transfer method based on single-view multi-plane image guidance, characterized by: The following steps are involved: Step 1: Construct multi-plane images MPIs; 1-1: Generate single-view multi-plane images MPIs from a single image using the Encoder-Decoder structure of a fully convolutional network. The Encoder uses the Resnet-50 model and the input is an image. , the output is the feature representation , the decoder design is similar to Monodepth2, and the input is the features output by the encoder and parallax values , the output is different depths Next Image, which contains RGB values and bulk density Next image, is the near plane, is the far plane; 1-2: Depth layer discretization; 1-3: Inference process, during inference, the Encoder runs only once to extract features, and the Decoder runs N times to generate Collection of images , each time corresponding to depth z, generate the entire MPIs; Step 2: Extract the style features of ink painting and define To provide a stylish ink painting image, is the content image at depth z; 2-1: Feature extraction, running VGG-19 network, from ink painting images Extracting ink painting style features , run the VGG-19 network N times, from the content image Extract N content features , get the content feature set ,make Representation characteristics At pixel position The eigenvector at ; 2-2: Nearest Neighbor Feature Matching (NNFM) Loss: Use NNFM loss to transfer the style stroke details of the ink painting to the content image to obtain a content image with style features. ; Step 3: view synthesis; 3-1: Predicting the 3D representation of the camera frustum and defining a content image with style features The coordinates of each pixel point on , using camera intrinsics Convert 2D pixel coordinates to 3D point coordinate expression: ; 3-2: Differentiable rendering, using differentiable rendering technology to render the reconstructed camera frustum into an image of the new view; 3-3: Render a new perspective image, render a camera rotation and pan New perspective image.

2. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: In step 1, the multi-plane image MPIs is a three-dimensional expression consisting of N planes parallel to a reference camera. These planes are located within the camera's viewing cone and are arranged at equal intervals in depth Z and disparity d.

3. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 2, characterized in that: Each of the Image is a 4-channel image representing the contents of a plane at depth Z, containing RGB values and bulk density ; in, The transpose symbol T represents a column vector. Refers to the three-dimensional real space. A three-dimensional vector is usually represented as [x, y, z]. represents the set of positive real numbers, that is, all real numbers greater than zero.

4. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 3, characterized in that: In step 1-2, the specific process of depth layer discretization is as follows: in the source camera's cone of view, define the depth range of the cone reconstruction ,in is the near plane, For the far plane, these values ​​are predefined hyperparameters that determine the depth perception range of the scene, which will be continuous depth range Discretize into N equally spaced intervals, each interval defines a depth layer, and each depth layer has a boundary value Used to represent the disparity value in different depth ranges, the boundary value of each interval Calculated by the following formula: in, is the minimum disparity value, is the maximum disparity value, and n is the index of the current depth layer.

5. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: In step 2-2: Use the nearest neighbor feature matching NNFM loss to separate the style stroke details of the ink painting from the ink style features Migrate to N depth-level content features respectively , which can be written as: where M is The number of pixels in Representation characteristics At pixel position The eigenvector at Representation characteristics At pixel position The eigenvector at For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space, defining the cosine distance is a vector and The cosine of the angle between is used to measure style features. and content features Similarity between: for For each feature of , minimize the cosine distance to its VGG-19 feature space in the style image The nearest neighbors in are used for feature matching.

6. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-1: From perspective 3D coordinates To Cartesian coordinates The conversion is : in, Represents the transformation from the camera coordinate system to the world coordinate system, is a constant representing the Layer and Distance between layers: Two-dimensional real number space to the set of positive real numbers The mapping is: 。 7. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-2 can be expressed as: in, is the rendered image, is the number of depth levels, From the first floor to the the cumulative transparency of the layer, and They represent j=1, i.e. the volume density value and RGB value of the first layer MPI, refers to the two-dimensional real number space, represents the set of positive real numbers, that is, all real numbers greater than zero.

8. The 3D ink painting style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-3 Rendering the RGB image at any new viewpoint requires additional steps: in For plane The normal vector of , then calculate the projection of the target camera ,in , Represents the Euclidean distance between two intersection points: After completing this step, , , Replace with , , , the new perspective image can be rendered using the differentiable rendering formula.

Images