3D ink style migration method based on single-view multi-plane image guidance

Abstract

The invention discloses a 3D ink style migration method based on single-view multi-plane image guidance, and the method comprises the following steps: 1, constructing a multi-plane image MPIs: 1-1: employing an Encoder-Decoder structure of a full convolution network to generate a single-view multi-plane image MPIs from a single image; 1-2, discretizing a depth layer; 1-3, a reasoning process: during reasoning, Encoder operates only once to extract features; 2, extracting style characteristics of the wash painting, defining Is as a wash painting image providing a style, and defining Izi as a content image with the depth of z; 2-1, feature extraction, 2-2, nearest neighbor feature matching NNFM loss, and 3, view synthesis. According to the method, the single-view multi-plane image (MPIs) can be better utilized to express the layered structure of the ink painting, the edge stroke effect and the clean blank leaving effect are greatly improved, meanwhile, the ink style migration effect is greatly improved by adopting the 3D ink style migration technical route guided by the single-view multi-plane image, and the 3D ink style migration efficiency is improved. And new visual angle synthesis of the ink-wash style scene is realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and in particular to a 3D ink style transfer method based on single-view multi-plane image guidance. Background Art

[0002] In the field of graphics and computer vision, image style transfer is a challenging task that aims to transfer the visual style of one image to another while retaining the semantic content of the original image. This technology has been widely used in many fields, including artistic creation, movie special effects, game design, etc.

[0003] Traditional image style transfer methods usually rely on optimization algorithms, such as the neural style transfer method proposed by Gatys et al. These methods achieve image style transfer by minimizing the content loss and style loss between the original image and the target style image. However, these methods are often inefficient when processing large-scale images and complex scenes, and require a lot of computing resources and time.

[0004] With the development of deep learning technology, image style transfer methods based on deep neural networks have gradually become mainstream. Deep learning models such as generative adversarial networks (GANs) and convolutional neural networks (CNNs) have been widely used in image style transfer tasks and have achieved better results. These methods usually include a generator network and a discriminator network, and achieve image style transfer through adversarial training.

[0005] However, most of the current image style transfer methods are performed on 2D images, and there are still challenges in stylized rendering of 3D scenes. Especially when applying the traditional ink painting style to 3D scenes, due to the unique characteristics of ink painting, such as brushstrokes, blank space, ink stain effects, etc., existing 3D style transfer methods often cannot effectively capture and express the unique characteristics of ink painting. Summary of the invention

[0006] The purpose of this invention is to transfer the style of ink painting to single-view multi-plane images MPIs by combining deep learning and traditional image processing technology, which contains RGB-σ images of multiple planes, each of which expresses the content of the scene at a certain depth. It aims to better express the hierarchical structure characteristics of ink painting, constrain the edges of objects at different spatial levels, and improve the effect of edge strokes and clean white space. Secondly, NNFM is used to use effective features to improve the style transfer effect of the model. Finally, the three-dimensional expression of the camera cone is predicted, and the differentiable rendering technology is used to render a new perspective image with ink style.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A 3D ink style transfer method based on single-view multi-plane image guidance includes the following steps:

[0009] Step 1: construct multi-plane images MPI s;

[0010] 1-1: Use the Encoder-Decoder structure of the fully convolutional network to generate a single-view multi-plane image MPI s from a single image. The Encoder uses the Resnet-50 model and the input is image I c , the output is the feature representation F c , the decoder design is similar to Monodepth2, and the input is the feature F output by the encoder c and parallax value The output is different depths z∈[z f ,z g ], which contains the RGB value c z and the volume density σ z ;

[0011] 1-2: Depth layer discretization;

[0012] 1-3: Inference process, during inference, the Encoder runs only once to extract features, and the Decoder runs N times to generate a set of RGB-σ images Each time corresponding to the depth z, generate the entire MPI s;

[0013] Step 2: Extract the style features of ink painting and define I s To provide a stylish ink painting image, is the content image at depth z;

[0014] 2-1: Feature extraction, running VGG-19 network, from ink painting image I s Extract the ink painting style feature F s , run the VGG-19 network N times, from the content image Extract N content features from Get content feature set make Representation characteristics The feature vector at pixel position (i, j);

[0015] 2-2: Nearest neighbor feature matching NNFM loss: Use NNFM loss to transfer the style stroke details of the ink painting to the content image to obtain the content image with style features

[0016] Step 3: view synthesis;

[0017] 3-1: Predicting the 3D representation of the camera frustum and defining a content image with style features The coordinates of each pixel on the T ∈R 2 , using the camera intrinsic parameter K∈R 3×3 Convert 2D pixel coordinates to 3D point coordinate expression: [x, y, z] T ;

[0018] 3-2: Differentiable rendering, using differentiable rendering technology to render the reconstructed camera frustum into an image of the new view;

[0019] 3-3: Render the new perspective image, render a camera rotation R∈R 3×3 and translation t∈R 3 New perspective image.

[0020] Preferably, in step 1, the multi-plane image MPIs is a three-dimensional expression consisting of N planes parallel to a reference camera, wherein the planes are located within the camera frustum and are arranged at equal intervals in depth Z and parallax d.

[0021] Preferably, each of the RGB-σ images is a 4-channel image, used to represent the planar content at a depth of Z, including RGB values ​​c z :[x,y] T →R 3 and the volume density σ z :[x,y] T →R + ;

[0022] Where [x,y] T The transpose symbol T represents a column vector, R 3 Refers to the three-dimensional real space. A three-dimensional vector is usually represented as [x, y, z], R + represents the set of positive real numbers, that is, all real numbers greater than zero.

[0023] Preferably, in step 1-2, the specific process of discretizing the depth layer is as follows: in the source camera's view cone, define the depth range of the view cone reconstruction [z f ,z g ], where z f is the near plane, z g These values ​​are predefined hyperparameters that determine the depth perception range of the scene, converting the continuous depth range [z f ,z g ] is discretized into N equally spaced intervals, each of which defines a depth layer, and each depth layer has a boundary value Used to represent the disparity value in different depth ranges, the boundary value d of each interval nCalculated by the following formula:

[0024] d n =d f +(n-1) / N·(d g -d f )

[0025] Among them, d f =1 / z f is the minimum disparity value, d g =1 / z g is the maximum disparity value and n is the index of the current depth layer.

[0026] Preferably, in step 2-2: using the nearest neighbor feature matching NNFM loss to extract the style brushstroke details of the ink painting from the ink style feature F s Migrate to N depth levels of content features respectively It can be written as:

[0027]

[0028] Where M is The number of pixels in Representation characteristics The feature vector at pixel position (i, j), F s (i',j') represents feature F s The feature vector at the pixel position (i', j'). For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space and define the cosine distance D(v 1 ,v 2 ) is the vector v 1 and v 2 The cosine value of the angle between is used to measure the style feature F s and content features The similarity between:

[0029]

[0030] for For each feature of the style image, we minimize the cosine distance to its VGG-19 feature space F s The nearest neighbors in are used for feature matching.

[0031] Preferably, step 3-1: from the perspective three-dimensional coordinates [x, y, z] T To Cartesian coordinates [X,Y,Z] T The conversion of is ζ(·):

[0032]

[0033] Among them, ζ(·) represents the transformation from the camera coordinate system to the world coordinate system, is a constant representing the distance between the i-th layer and the i+1-th layer:

[0034] Two-dimensional real number space R 2 To the set of positive real numbers R + The mapping is:

[0035]

[0036] Preferably, step 3-2 can be expressed as:

[0037]

[0038] in, is the rendered image, N is the number of depth levels, T i is the cumulative transparency from the first layer to the i-th layer, σ zj and δ zj They represent j=1, i.e., the volume density value and RGB value of the first layer MPI, R 2 refers to the two-dimensional real number space, R + represents the set of positive real numbers, that is, all real numbers greater than zero.

[0039] Preferably, step 3-3 requires additional steps to render the RGB image at any new viewing angle:

[0040]

[0041] Where m = [0,0,1] T For plane The normal vector of Then calculate the projection of the target camera in

[0042] Represents the Euclidean distance between two intersection points:

[0043]

[0044] After completing this step, replace c, σ, δ with c', σ', δ' respectively, and you can render the new perspective image using the differentiable rendering formula.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention can better utilize single-view multi-plane images (MPIs) to express the layered structure of ink painting, greatly improving the effect of edge strokes and clean white space. At the same time, by adopting the 3D ink style transfer technology route guided by single-view multi-plane images, the ink style transfer effect is greatly improved, and a new perspective synthesis of ink style scenes is achieved.

[0047] The present invention predicts the three-dimensional expression of the camera cone through a single-view multi-plane image guidance method, and uses this three-dimensional expression to give the relative position and angle change of the target camera relative to the source camera in the three-dimensional space, thereby achieving the purpose of efficiently rendering the RGB image under the target camera view. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The technical roadmap of 3D ink style transfer guided by single-view multi-plane images in this invention;

[0049] Figure 2 The nearest neighbor feature matching NNFM style stroke detail migration map in the present invention.

[0050] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] The background of the solution proposed in this application is:

[0053] With the widespread application of visual content creation in mobile photography and augmented reality / virtual reality (AR / VR), traditional image forms have been unable to meet the growing practical needs. Especially with the increasing demand for creating 3D content from 2D image assets, there is an urgent need to study a method that can expand traditional ink painting to dimensions beyond the 2D image plane.

[0054] In the 3D stylization problem, ensuring that the stylized result is perceptually similar to the style of a given reference image is a key challenge. Existing 3D scene style transfer methods are mostly based on Western painting styles. Ink paintings have significant differences in visual features, semantic features, and aesthetic criteria from photographs and oil paintings. Existing 3D style transfer methods usually transfer the overall style of the entire scene, which is contrary to the use of outline blanking and other techniques in ink painting style to reflect the characteristics of objects at different levels. As a result, these methods cannot achieve a clean blanking effect and it is difficult to accurately capture the unique brushstroke characteristics of ink paintings. Therefore, it is necessary to further explore 3D style transfer methods targeting the characteristics of Chinese ink paintings in order to better preserve and express the artistic characteristics of ink paintings.

[0055] Example

[0056] like Figure 1 and Figure 2 As shown, the present application provides a 3D ink style transfer method based on single-view multi-plane image guidance, comprising the following steps:

[0057] Step 1: Construct multi-plane images MPIs. Multi-plane images MPIs are a three-dimensional expression consisting of N planes parallel to the reference camera. These planes are located in the camera frustum and are arranged at equal intervals at depth Z and disparity d. Each of the RGB-σ images is a 4-channel image used to represent the plane content at depth Z, containing RGB values ​​c z :[x,y] T →R 3 and the volume density σ z :[x,y] T →R + ;

[0058] Where [x,y] T The transpose symbol T represents a column vector, R 3 Refers to the three-dimensional real space. A three-dimensional vector is usually represented as [x, y, z]. + represents the set of positive real numbers, that is, all real numbers greater than zero.

[0059] 1-1: Generate single-view multi-plane images MPIs from a single image using the Encoder-Decoder structure of a fully convolutional network. The Encoder uses the Resnet-50 model and the input is image I c , the output is the feature representation F c , the decoder design is similar to Monodepth2, and the input is the feature F output by the encoder c and parallax value The output is different depths z∈[z f ,z g ], which contains the RGB value c z and the volume density σ z ;

[0060] 1-2: Depth layer discretization. The specific process of depth layer discretization is as follows: In the source camera's cone of view, define the depth range of the cone reconstruction [z f ,z g ], where z f is the near plane, z g These values ​​are predefined hyperparameters that determine the depth perception range of the scene, converting the continuous depth range [z f ,z g] is discretized into N equally spaced intervals, each of which defines a depth layer, and each depth layer has a boundary value Used to represent the disparity value in different depth ranges, the boundary value d of each interval n Calculated by the following formula:

[0061] d n =d f +(n-1) / N·(d g -d f )

[0062] Among them, d f =1 / z f is the minimum disparity value, d g =1 / z g is the maximum disparity value and n is the index of the current depth layer.

[0063] 1-3: Inference process, during inference, the Encoder runs only once to extract features, and the Decoder runs N times to generate a set of RGB-σ images Each time corresponding to the depth z, generate the entire MPI s;

[0064] Step 2: Extract the style features of ink painting and define I s To provide a stylish ink painting image, is a content image with depth z, using a VGG-19 network pre-trained on the ImageNet dataset. The network consists of 5 layer blocks: conv1, conv2, conv3, conv4, conv5. This application uses the conv3 block as a feature extractor.

[0065] 2-1: Feature extraction, running VGG-19 network, from ink painting image I s Extract the ink painting style feature F s , run the VGG-19 network N times, from the content image Extract N content features from Get content feature set make Representation characteristics The feature vector at pixel position (i, j);

[0066] 2-2: Nearest neighbor feature matching NNFM loss: Use NNFM loss to transfer the style stroke details of the ink painting to the content image to obtain the content image with style features The nearest neighbor feature matching NNFM loss is used to extract the style stroke details of the ink painting from the ink style feature F s Migrate to N depth levels of content features respectively By minimizing the NNFM loss, the style transfer effect of each MPI plane is adjusted to effectively capture and express the unique characteristics of ink painting, which can be written as:

[0067]

[0068] Where M is The number of pixels in Representation characteristics The feature vector at pixel position (i, j), F s (i',j') represents feature F s The feature vector at the pixel position (i', j'). For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space and define the cosine distance D(v 1 ,v 2 ) is the vector v 1 and v 2 The cosine value of the angle between is used to measure the style feature F s and content features The similarity between:

[0069]

[0070] for For each feature of the style image, we minimize the cosine distance to its VGG-19 feature space F s The nearest neighbors in are used for feature matching.

[0071] Step 3-1: From perspective 3D coordinates [x, y, z] T To Cartesian coordinates [X,Y,Z] T The conversion of is ζ(·):

[0072]

[0073] Among them, ζ(·) represents the transformation from the camera coordinate system to the world coordinate system, is a constant representing the distance between the i-th layer and the i+1-th layer:

[0074] Two-dimensional real number space R 2 To the set of positive real numbers R + The mapping is:

[0075]

[0076] Step 3: view synthesis;

[0077] 3-1: Predicting the 3D representation of the camera frustum and defining a content image with style features The coordinates of each pixel on the T ∈R 2 , using the camera intrinsic parameter K∈R 3×3 Convert 2D pixel coordinates to 3D point coordinate expression: [x, y, z] T ;

[0078] 3-2: Differentiable rendering, using differentiable rendering technology to render the reconstructed camera frustum into an image of the new view, which can be expressed as:

[0079]

[0080] in, is the rendered image, N is the number of depth levels, T i is the cumulative transparency from the first layer to the i-th layer, σ zj and δ zj They represent j=1, i.e., the volume density value and RGB value of the first layer MPI, R 2 refers to the two-dimensional real number space, R + represents the set of positive real numbers, that is, all real numbers greater than zero.

[0081] 3-3: Render the new perspective image, render a camera rotation R∈R 3×3 and translation t∈R 3 New perspective images, rendering RGB images under any new perspective requires additional steps:

[0082]

[0083] Where m = [0,0,1] T For plane The normal vector of Then calculate the projection of the target camera in

[0084] Represents the Euclidean distance between two intersection points:

[0085]

[0086] After completing this step, replace c, σ, δ with c', σ', δ' respectively, and you can render the new perspective image using the differentiable rendering formula.

[0087] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of more restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of the same element in the process, device, article or method including the element.

[0088] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A 3D ink style transfer method based on single-view multi-plane image guidance, characterized in that: The following steps are involved: Step 1: construct multi-plane images MPIs; 1-1: Generate single-view multi-plane images MPIs from a single image using the Encoder-Decoder structure of a fully convolutional network. The Encoder uses the Resnet-50 model and the input is image I c , the output is the feature representation F c , the decoder design is similar to Monodepth2, and the input is the feature F output by the encoder c and parallax value The output is different depths z∈[z f ,z g ], which contains the RGB value c z and the volume density σ z ; 1-2: Depth layer discretization; 1-3: Inference process, during inference, the Encoder runs only once to extract features, and the Decoder runs N times to generate a set of RGB-σ images Each time corresponding to depth z, generate the entire MPIs; Step 2: Extract the style features of ink painting and define I s To provide a stylish ink painting image, is the content image at depth z; 2-1: Feature extraction, running VGG-19 network, from ink painting image I s Extract the ink painting style feature F s , run the VGG-19 network N times, from the content image Extract N content features from Get content feature set make Representation characteristics The feature vector at pixel position (i, j); 2-2: Nearest neighbor feature matching NNFM loss: Use NNFM loss to transfer the style stroke details of the ink painting to the content image to obtain the content image with style features Step 3: view synthesis; 3-1: Predicting the 3D representation of the camera frustum and defining a content image with style features The coordinates of each pixel on the T ∈R 2 , using the camera intrinsic parameter K∈R 3×3 Convert 2D pixel coordinates to 3D point coordinates: [x, y, z] T ; 3-2: Differentiable rendering, using differentiable rendering technology to render the reconstructed camera frustum into an image of the new view; 3-3: Render a new perspective image, render a camera rotation R∈R 3×3 and translation t∈R 3 New perspective image.

2. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: In the step 1, the multi-plane image MPIs is a three-dimensional expression consisting of N planes parallel to the reference camera. These planes are located in the camera frustum and are arranged at equal intervals in depth Z and disparity d.

3. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 2, characterized in that: Each of the RGB-σ images is a 4-channel image used to represent the planar content at depth Z, containing RGB values ​​c z :[x,y] T →R 3 and the volume density σ z :[x,y] T →R + ; Where [x,y] T The transpose symbol T represents a column vector, R 3 Refers to the three-dimensional real space. A three-dimensional vector is usually represented as [x, y, z]. + represents the set of positive real numbers, that is, all real numbers greater than zero.

4. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 3, characterized in that: In step 1-2, the specific process of depth layer discretization is as follows: in the source camera's view cone, define the depth range of the view cone reconstruction [z f ,z g ], where z f is the near plane, z g These values ​​are predefined hyperparameters that determine the depth perception range of the scene, converting the continuous depth range [z f ,z g ] is discretized into N equally spaced intervals, each of which defines a depth layer, and each depth layer has a boundary value Used to represent the disparity values ​​in different depth ranges, the boundary value d of each interval n Calculated by the following formula: d n =d f +(n-1) / N·(d g -d f ) Among them, d f =1 / z f is the minimum disparity value, d g =1 / z g is the maximum disparity value and n is the index of the current depth layer.

5. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: In step 2-2: Use the nearest neighbor feature matching NNFM loss to extract the style stroke details of the ink painting from the ink style feature F s Migrate to N depth levels of content features respectively It can be written as: Where M is The number of pixels in Representation characteristics The feature vector at pixel position (i, j), F s (i',j') represents feature F s The feature vector at the pixel position (i', j') is For each feature in , we minimize its cosine distance to the nearest neighbor in the VGG-19 feature space and define the cosine distance D(v1,v2) as the cosine value of the angle between vectors v1 and v2, which is used to measure the style feature F s and content features The similarity between: for For each feature of the style image, we minimize the cosine distance to its VGG-19 feature space F s The nearest neighbors in are used for feature matching.

6. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-1: From perspective 3D coordinates [x, y, z] T To Cartesian coordinates [X,Y,Z] T The conversion of is ζ(·): Among them, ζ(·) represents the transformation from the camera coordinate system to the world coordinate system, is a constant representing the distance between the i-th layer and the i+1-th layer: Two-dimensional real number space R 2 To the set of positive real numbers R + The mapping is:

7. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-2 can be expressed as: in, is the rendered image, N is the number of depth levels, T i is the cumulative transparency from the first layer to the i-th layer, σ zj and δ zj They represent j=1, i.e., the volume density value and RGB value of the first layer MPI, R 2 refers to the two-dimensional real number space, R + represents the set of positive real numbers, that is, all real numbers greater than zero.

8. The 3D ink style transfer method based on single-view multi-plane image guidance according to claim 1, characterized in that: Step 3-3 Rendering the RGB image at any new viewpoint requires additional steps: Where m = [0,0,1] T For plane The normal vector of Then calculate the projection of the target camera in Represents the Euclidean distance between two intersection points: After completing this step, replace c, σ, δ with c', σ', δ' respectively, and you can render the new perspective image using the differentiable rendering formula.

Images