A single-view image-driven neural rendering method for highly realistic virtual clothing

Abstract

The present invention relates to a highly realistic virtual clothing neural rendering method driven by a single-view image, and belongs to the field of computer vision and graphics images. First, a segmented clothing image, a clothing component mask, and a three-dimensional clothing mesh under an input posture are obtained; then, clothing feature encoding is performed on the segmented clothing image, the three-dimensional clothing mesh, and the clothing component mask to obtain style features, spatial features, and mapping features respectively; then, through a constructed clothing feature decoding module based on a neural radiation field, the style, space, and mapping features are integrated to obtain the color and density of the clothing neural radiation field used for rendering; finally, by solving the transformation relationship of rendering parameters of clothing query points between the observation space, the specification space, and the posture space, clothing neural rendering is performed, and finally the clothing rendering result is obtained. The present invention can be used to effectively represent clothing in a single-view image, and perform highly realistic virtual clothing rendering based on the input perspective and posture.

Description

Technical Field

[0001] The invention relates to a single-view image-driven high-realistic virtual clothing neural rendering method, and belongs to the field of computer vision and graphic images. Background Art

[0002] As an important virtual asset in modern digital society, virtual clothing occupies a pivotal position in the clothing design, sports, film, animation, game and other industries. The diversity of virtual clothing in patterns, layers and styles increases the difficulty of obtaining them. In addition, the interaction between clothing and the human body and the rendering effects of different materials also increase the challenge of accurate and realistic presentation of virtual clothing in specific scenes. Among the known methods, traditional texture mapping based on three-dimensional grids is difficult to handle high dynamic range and complex light and shadow effects, especially the performance on transmissive or reflective materials is often not realistic enough. Although the implicit function method can display clothing details with infinite resolution, it needs to solve a large number of optimization problems for intensive sampling calculations, and it relies on global function representation and has poor real-time interaction capabilities. Neural radiation field rendering effectively handles the transparency and reflection characteristics of materials by simulating the propagation of light in the scene and its complex interaction with the surface of objects. It can capture the subtle texture on clothing and present complex light and shadow effects, thereby producing high-quality and realistic images. In addition, it supports rendering from continuously changing perspectives, which further enhances the coherence and realism of visual effects. For example, Wang (<Computer Vision and Pattern Recognition> ,2023) introduces a novel geometry-guided three-plane representation, which significantly enhances the robustness of feature representation. Weng (<Computer Vision and Pattern Recognition> ,2024) complements the missing details of the rendering and refines the texture by adding dense views generated by the conditional diffusion model.<IEEE Transactions on Pattern Analysis and Machine Intelligence> ,2022) realized the feature extraction of self-attention mechanism, which enhanced the global context perception ability and detail expression. The above-mentioned known methods all render the dressed human body from a single image, multi-view image or video input of the dressed human body.

[0003] Since real clothing often contains different structures, layers and texture features, and the clothing in the input data often lacks overall topology and detailed texture information due to being blocked by the human body or the line of sight, the existing known methods fail to consider the situation where clothing information is missing, and separate the complete clothing information from the single-view human body image and achieve high-quality neural rendering. Therefore, the present invention proposes a high-realism virtual clothing neural rendering method driven by a single-view image. The mapping relationship between the input clothing image, clothing three-dimensional mesh, and clothing mask point cloud is feature encoded and the overall structure, local details, and mapping information of the clothing are extracted to solve the problems of topological structure distortion, detail blur and artifacts; the transformation relationship of the rendering parameters of the clothing query point between the observation space, the standard space, and the posture space is solved in combination with the human body posture parameters, and the image of the clothing at the specified perspective and posture is rendered to solve the continuous stability of the virtual clothing at different perspectives and postures. Summary of the invention

[0004] The present invention provides a high-fidelity virtual clothing neural rendering method driven by a single-view image, which is used to solve the problems of rendering topological distortion, blurred details and poor stability of virtual clothing generated based on single-view images, and generate high-fidelity clothing rendering images.

[0005] The technical solution of the present invention is: a high-realistic virtual clothing neural rendering method driven by a single-view image, comprising the following steps:

[0006] Step 1: Reconstruct the 3D clothing from the input single-view human image I based on clothing parts and human posture, and obtain the clothing image I after component segmentation. g , clothing parts mask m g and the 3D clothing mesh M under the input pose g ;

[0007] Step 2: Clothing image I after component segmentation g , 3D clothing mesh M g , clothing parts mask m g To point cloud P g The mapping relationship P g →m g Encode clothing features to obtain style features F style , spatial feature F space and mapping feature F mapping ;

[0008] Step 3: By constructing a clothing feature decoding module based on neural radiation field, the style, space and mapping features are integrated to obtain the clothing query point color c in the standard space for neural radiation field rendering. c With density σ c ;

[0009] Step 4: Perform clothing neural rendering by solving the transformation relationship of rendering parameters between the clothing query point in the observation space, standard space and posture space, and finally obtain a high-realistic clothing rendering result I r .

[0010] Furthermore, the specific process of Step 2 is as follows:

[0011] Step 2.1 Clothing style feature encoding

[0012] First, based on the clothing image I after component segmentation g , by constructing the clothing style feature encoder f z Compression encoding is performed, and I g Mapped to a latent vector of an intermediate layer

[0013] Then, based on the generative adversarial network StyleGAN, the mapping network C is defined m , maps the input latent vector z to another latent vector ws used to control the subsequent clothing style and content encoding, and then passes through the constructed style synthesis network C sys , ws in C sys is compiled into a three-plane feature representation Plane Tri It is a three-plane feature map with 32 channels, height and width of 256.

[0014] Finally, define the sampling function F s , according to the 3D coordinates of the Plane Tri Perform orthogonal projection sampling to obtain N s Feature points, through the constructed clothing image style encoding network: Finally, we get the clothing style feature F style .

[0015] Step 2.2 Clothing spatial feature extraction

[0016] First, stitch the clothing point cloud set to obtain the clothing point cloud of the observed person where {P1,P2,...,P i} is the clothing point cloud set of each part generated in Step 1. According to the camera parameters, P g Along the observation angle optical path d obs Projection to obtain the mapping UV g ;

[0017] Then, extract the clothing image I g The feature map F g , according to the mapping map UV gThe index pair F g and I g Perform 2D to 3D feature mapping to obtain the clothing mesh vertex features F in the observation space ver and color feature F rgb , splicing F ver and F rgb And through convolution, dimensionality reduction is performed to obtain low-level 3D features of clothing in the observation space;

[0018] Finally, the sparse convolutional network is used to convolve and downsample the low-level 3D features of clothing layer by layer to obtain the clothing spatial features F space .

[0019] Step 2.3 Clothing mapping feature encoding

[0020] First, we create a point cloud P containing clothing g The sampling space is calculated by calculating the query point q in the sampling space s Distance P g The Euclidean distance of the nearest point in the equation is selected to satisfy ||q s -p g ||2≤d t Q s As sampling point, point p g ∈P g , d t is the sampling threshold;

[0021] Then the sampling point set Q in the observation space is s Along the observation angle optical path d obs Perform projection transformation and map it into clothing mask image m g Projection mask UV on s , according to UV s The position encoding pair I g Perform pixel sampling and feature sampling on the feature map extracted by the 2D encoder to obtain the sampled pixel mapping feature F p and the sampled pixel color feature F c ;

[0022] Finally, the sampled pixel map feature F is fused p and the sampled pixel color feature F c , get the clothing mapping feature F mapping .

[0023] Furthermore, the specific process of Step 3 is as follows:

[0024] First, by analyzing the style feature F style , spatial feature F space and mapping feature F mapping Fusion is performed to generate the fusion feature F of clothing fus ;

[0025] Then, the clothing feature decoding module based on neural radiation field is used to decode the fusion feature F fus Decode clothing features by defining a clothing feature decoding module D based on neural radiation field Nerf :(F fus ,q c ,d obs )→(c c ,σ c ), where q c and d obs is the query point and observation light path in the clothing specification space, and finally obtains the clothing query point color c in the specification space for neural radiation field rendering c With density σ c .

[0026] Furthermore, the specific process of step 4 is as follows:

[0027] First, the query point q on the clothing in the target posture space t By inverse transformation to a point in canonical space in, and normative space clothing point cloud Points in There is the closest Euclidean distance;

[0028] Then define and In the specified observation optical path d t With the same color and density field function f Nerf ,Right now Specify the observation optical path d t The clothing query point q in the target pose space under t The color of the neural radiation field c t With density σ t It is expressed as follows: D Nerf (q t ,d t ,F style ,F Space ,F Mapping )→(c t ,σ t );

[0029] Finally, based on the color, transparency and rendering parameters of the neural radiation field, the spatial voxel integration is performed to calculate the two-dimensional image pixel I of the clothing neural rendering. r (r) as follows: where r(t) = o + td t It represents the direction from the camera optical center o along d t The reverse path ray in the direction, t u and tl is the upper and lower bounds of the propagation of r(t) in the clothing sampling area. Volume density σ t (r(t)) represents the particle density of ray r(t) at point t, and the voxel color value c t (r(t),d t ) is represented by the ray r(t) along d t The color value at direction t. Where T(t) represents the ray from the lower bound t l The cumulative transmittance of the propagation process to t. r (r) is the final clothing rendering result.

[0030] The beneficial effects of the present invention are:

[0031] 1. The known methods mainly reconstruct, render and learn the dressed characters based on the overall features, without considering the structure and texture priors of the clothing itself. When the input clothing lacks some information, it will cause the rendering results to be distorted, blurred and have artifacts. The present invention uses a clothing feature encoder to learn the topology and texture priors of clothing separately, extract clothing style, space and mapping features, and realize clothing neural radiation field rendering. The generated results have realistic topological structures and texture details, thereby improving the rendering realism of clothing.

[0032] 2. The known methods mainly realize rendering of a single perspective and posture based on the dressed human body, and do not consider the rendering robustness and stability of virtual clothing under arbitrary perspective and posture changes. The present invention calculates the clothing transformation relationship between different posture spaces, outputs the neural radiation field color and density value of the clothing query point in the target space, and improves the rendering quality of clothing under different perspectives and postures. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 This is a rendering result diagram of the clothing of the present invention. DETAILED DESCRIPTION

[0035] Example 1: Figure 1 As shown, a clothing reconstruction and rendering method for a dressed human body image includes the following steps:

[0036] Step 1: Reconstruct the 3D clothing from the input single-view human image I based on clothing parts and human posture, and obtain the clothing image I after component segmentation. g , clothing parts mask m g and the 3D clothing mesh M under the input pose g ;

[0037] Step 2: Clothing image I after component segmentation g , 3D clothing mesh M g , clothing parts mask m g To point cloud P g The mapping relationship P g →m g Encode clothing features to obtain style features F style , spatial feature F space and mapping feature F mapping ;

[0038] Step 3: By constructing a clothing feature decoding module based on neural radiation field, the style, space and mapping features are integrated to obtain the clothing query point color c in the standard space for neural radiation field rendering. c With density σ c ;

[0039] Step 4: Perform clothing neural rendering by solving the transformation relationship of rendering parameters between the clothing query point in the observation space, standard space and posture space, and finally obtain a high-realistic clothing rendering result I r .

[0040] The specific process of Step 2 is as follows:

[0041] Step 2.1 Clothing style feature encoding

[0042] First, based on the clothing image I after component segmentation g , by constructing the clothing style feature encoder f z Compression encoding is performed, and I g Mapped to a latent vector of an intermediate layer

[0043] Then, based on the generative adversarial network StyleGAN, the mapping network C is defined m , maps the input latent vector z to another latent vector ws used to control the subsequent clothing style and content encoding, and then passes through the constructed style synthesis network C sys , ws in C sys is compiled into a three-plane feature representation Plane Tri It is a three-plane feature map with 32 channels, height and width of 256.

[0044] Finally, define the sampling function F s , according to the 3D coordinates of the Plane Tri Perform orthogonal projection sampling to obtain N s Feature points, through the constructed clothing image style encoding network: Finally, we get the clothing style feature Fstyle .

[0045] Step 2.2 Clothing spatial feature extraction

[0046] First, stitch the clothing point cloud set to obtain the clothing point cloud of the observed person where {P1,P2,...,P i} is the clothing point cloud set of each part generated in Step 1. According to the camera parameters, P g Along the observation angle optical path d obs Projection to obtain the mapping UV g ;

[0047] Then, extract the clothing image I g The feature map F g , according to the mapping map UV g The index pair F g and I g Perform 2D to 3D feature mapping to obtain the clothing mesh vertex features F in the observation space ver and color feature F rgb , splicing F ver and F rgb And through convolution, dimensionality reduction is performed to obtain low-level 3D features of clothing in the observation space;

[0048] Finally, the sparse convolutional network is used to convolve and downsample the low-level 3D features of clothing layer by layer to obtain the clothing spatial features F space .

[0049] Step 2.3 Clothing mapping feature encoding

[0050] First, we create a point cloud P containing clothing g The sampling space is calculated by calculating the query point q in the sampling space s Distance P g The Euclidean distance of the nearest point in the equation is selected to satisfy ||q s -p g ||2≤d t Q s As sampling point, point p g ∈P g , d t is the sampling threshold;

[0051] Then the sampling point set Q in the observation space is s Along the observation angle optical path d obs Perform projection transformation and map it into clothing mask image m g Projection mask UV on s , according to UV s The position encoding pair I gPerform pixel sampling and feature sampling on the feature map extracted by the 2D encoder to obtain the sampled pixel mapping feature F p and the sampled pixel color feature F c ;

[0052] Finally, the sampled pixel map feature F is fused p and the sampled pixel color feature F c , get the clothing mapping feature F mapping .

[0053] The specific process of Step 3 is as follows:

[0054] First, by analyzing the style feature F style , spatial feature F space and mapping feature F mapping Fusion is performed to generate the fusion feature F of clothing fus ;

[0055] Then, the clothing feature decoding module based on neural radiation field is used to decode the fusion feature F fus Decode clothing features by defining a clothing feature decoding module D based on neural radiation field Nerf :(F fus ,q c ,d obs )→(c c ,σ c ), where q c and d obs is the query point and observation light path in the clothing specification space, and finally obtains the clothing query point color c in the specification space for neural radiation field rendering c With density σ c .

[0056] Furthermore, the specific process of Step 4 is as follows:

[0057] First, the query point q on the clothing in the target posture space t By inverse transformation to a point in canonical space in, and normative space clothing point cloud Points in There is the closest Euclidean distance;

[0058] Then define and In the specified observation optical path d t With the same color and density field function f Nerf ,Right now Specify the observation optical path d t The clothing query point q in the target pose space under t The color of the neural radiation field ct With density σ t It is expressed as follows: D Nerf (q t ,d t ,F style ,F Space ,F Mapping )→(c t ,σ t );

[0059] Finally, based on the color, transparency and rendering parameters of the neural radiation field, the spatial voxel integration is performed to calculate the two-dimensional image pixel I of the clothing neural rendering. r (r) as follows: where r(t) = o + td t It represents the direction from the camera optical center o along d t The reverse path ray in the direction, t u and t l is the upper and lower bounds of the propagation of r(t) in the clothing sampling area. Volume density σ t (r(t)) represents the particle density of ray r(t) at point t, and the voxel color value c t (r(t),d t ) is represented by the ray r(t) along d t The color value at direction t. Where T(t) represents the ray from the lower bound t l The cumulative transmittance of the propagation process to t. r (r) is the final clothing rendering result.

[0060] Example 2: Figure 1-Figure 2 As shown, a clothing reconstruction and rendering method for a dressed human body image includes the following steps:

[0061] Step 1: Reconstruct the 3D clothing from the input single-view human image I based on clothing parts and human posture, and obtain the clothing image I after component segmentation. g , clothing parts mask m g and the 3D clothing mesh M under the input pose g .

[0062] Step 2.1 Clothing style feature encoding

[0063] First, based on the clothing image I after component segmentation g , by constructing the clothing style feature encoder f z Compression encoding, f z The residual network Resnet18 is used as the backbone, which includes convolutional layers, normalization layers, activation layers, several residual blocks and pooling layers. g Mapped to a latent vector of an intermediate layer

[0064] Then, based on the generative adversarial network StyleGAN, the mapping network C is defined m , maps the input latent vector z to another latent vector ws used to control the subsequent clothing style and content encoding, and then passes through the constructed style synthesis network C sys , ws in C sys is compiled into a three-plane feature representation Plane Tri It is a three-plane feature map with 32 channels, height and width of 256.

[0065] Finally, define the sampling function F s , the sampling mode is set to bilinear interpolation sampling, and the Plane is sampled according to the 3D coordinates Tri Perform orthogonal projection sampling to obtain N s Feature points, through the constructed clothing image style encoding network: Finally, we get the clothing style feature F style .

[0066] Step 2.2 Clothing spatial feature extraction

[0067] First, stitch the clothing point cloud set to obtain the clothing point cloud of the observed person where {P1,P2,...,P i} is the clothing point cloud set of each part generated in Step 1. According to the camera parameters, P g Along the observation angle optical path d obs Projection to obtain the mapping UV g ;

[0068] Then, clothing image I is extracted through Resnet18 g The feature map F g , according to the mapping map UV g The index pair F g and I g Perform 2D to 3D feature mapping to obtain the clothing mesh vertex features F in the observation space ver and color feature F rgb , splicing F ver and F rgb And through convolution, dimensionality reduction is performed to obtain low-level 3D features of clothing in the observation space;

[0069] Finally, the sparse convolutional network is used to convolve and downsample the low-level 3D features of clothing layer by layer to obtain the clothing spatial features F space .

[0070] Step 2.3 Clothing mapping feature encoding

[0071] First, we create a point cloud P containing clothing g The sampling space along the observation angle optical path d obs Several sampling beams are emitted, and 48 sampling points are set on each beam. The query point q in the sampling space is calculated. s Distance P g The Euclidean distance of the nearest point in the equation is selected to satisfy ||q s -p g ||2≤d t Q s As sampling point, point p g ∈P g , sampling threshold d t Set to 0.0025;

[0072] Then the sampling point set Q in the observation space is s Along the observation angle optical path d obs Perform projection transformation and map it into clothing mask image m g Projection mask UV on s , according to UV s The position encoding pair I g Perform pixel sampling and feature sampling on the feature map extracted by the 2D encoder to obtain the sampled pixel mapping feature F p and the sampled pixel color feature F c ;

[0073] Finally, the sampled pixel map feature F is fused p and the sampled pixel color feature F c , get the clothing mapping feature F mapping .

[0074] The specific process of Step 3 is as follows:

[0075] First, concatenate the style features F style , spatial feature F space and mapping feature F mapping It is a 96-channel concatenated feature, and after the convolution layer with a convolution kernel size of 1, the dimension is reduced and the output is an intermediate feature of 32 channels.

[0076] Then, the intermediate features pass through a Transformer feature fusion module, which contains multiple Transformer layers. Each Transformer layer consists of a multi-head self-attention mechanism and a feedforward neural network, which is realized through residual block connection and normalization layer. The multi-head attention mechanism contains 3 attention heads, and the feature dimension of each attention head is 16. The dimension of the intermediate layer of the feedforward neural network is 32, generating the fusion feature F of the clothing. fus ;

[0077] Then, the clothing feature decoding module based on neural radiation field is used to decode the fusion feature F fus Decode clothing features by defining a clothing feature decoding module D based on neural radiation field Nerf :(F fus ,q c ,d obs )→(c c ,σ c ), where D Nerf Receive the query point coordinates q on the garment in the canonical space c , Observation light path direction d obs , fusion feature F fus The color and density of the query point are predicted through a fully connected layer, a hidden layer with ReLU activation function, and an output layer with a fully connected layer and Sigmoid activation function, where a jump connection is added to the hidden layer. Finally, the color c of the clothing query point in the canonical space for neural radiation field rendering is obtained. c With density σ c .

[0078] The specific process of Step 4 is as follows:

[0079] First, the query point q on the clothing in the target posture space t By inverse transformation to a point in canonical space in, and normative space clothing point cloud Points in There is the closest Euclidean distance;

[0080] Then define and In the specified observation optical path d t With the same color and density field function f Nerf ,Right now Specify the observation optical path d t The clothing query point q in the target pose space under t The color of the neural radiation field c t With density σ t It is expressed as follows: D Nerf (q t ,d t ,F style ,F Space ,F Mapping )→(c t ,σ t );

[0081] Finally, based on the color, transparency and rendering parameters of the neural radiation field, the spatial voxel integration is performed to calculate the two-dimensional image pixel I of the clothing neural rendering. r(r) as follows: where r(t) = o + td t It represents the direction from the camera optical center o along d t The reverse path ray in the direction, t u and t l is the upper and lower bounds of the propagation of r(t) in the clothing sampling area. Volume density σ t (r(t)) represents the particle density of ray r(t) at point t, and the voxel color value c t (r(t),d t ) is represented by the ray r(t) along d t The color value at direction t. Where T(t) represents the ray from the lower bound t l The cumulative transmittance of the propagation process to t. r (r) is the final clothing rendering result. The clothing rendering results of different perspectives and postures are shown in the figure. Figure 2 As shown, it is shown that the clothing rendering results of the method of the present invention have high rendering quality and realism under different viewing angles and postures.

[0082] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A single-view image-driven neural rendering method for highly realistic virtual clothing. Features: The steps include: Step 1: Reconstruct the 3D clothing from the input single-view human image I based on clothing parts and human posture, and obtain the clothing image I after component segmentation. g , clothing parts mask m g and the 3D clothing mesh M under the input pose g ; Step 2: Clothing image I after component segmentation g , 3D clothing mesh M g , clothing parts mask m g To point cloud P g The mapping relationship P g →m g Encode clothing features to obtain style features F style , spatial feature F space and mapping feature F mapping ; Step 3: By constructing a clothing feature decoding module based on neural radiation field, the style, space and mapping features are integrated to obtain the clothing query point color c in the standard space for neural radiation field rendering. c With density σ c ; Step 4: Perform clothing neural rendering by solving the transformation relationship of rendering parameters between the clothing query point in the observation space, standard space and posture space, and finally obtain a high-realistic clothing rendering result I r ; The Step 2 includes: Step 2.1 Clothing style feature encoding First, based on the clothing image I after component segmentation g , by constructing the clothing style feature encoder f z Compression encoding is performed, and I g Mapped to a latent vector of an intermediate layer Then, based on the generative adversarial network StyleGAN, the mapping network C is defined m , maps the input latent vector z to another latent vector ws used to control the subsequent clothing style and content encoding, and then passes through the constructed style synthesis network C sys , ws in C sys is compiled into a three-plane feature representation Plane Tri It is a three-plane feature map with 32 channels, height and width of 256; Finally, define the sampling function F s , according to the 3D coordinates of the Plane Tri Perform orthogonal projection sampling to obtain N s Feature points, through the constructed clothing image style encoding network: Finally, we get the clothing style feature F style ; The Step 2 includes: Step 2.2 Clothing spatial feature extraction First, stitch the clothing point cloud set to obtain the clothing point cloud of the observed person where {P 1 ,P 2 ,...,P i } is the clothing point cloud set of each part generated in Step 1. According to the camera parameters, P g Along the observation angle optical path d obs Projection to obtain the mapping UV g ; Then, extract the clothing image I g The feature map F g , according to the mapping UV g The index pair F g and I g Perform 2D to 3D feature mapping to obtain the clothing mesh vertex features F in the observation space ver and color feature F rgb , splicing F ver and F rgb And through convolution, dimensionality reduction is performed to obtain low-level 3D features of clothing in the observation space; Finally, the sparse convolutional network is used to convolve and downsample the low-level 3D features of clothing layer by layer to obtain the clothing spatial features F space ; The Step 2 includes: Step 2.3 Clothing mapping feature encoding First, we create a point cloud P containing clothing g The sampling space is calculated by calculating the query point q in the sampling space s Distance P g The Euclidean distance of the nearest point in the equation is selected to satisfy ||q s -p g || 2 ≤d t Q s As sampling point, point p g ∈P g , d t is the sampling threshold; Then the sampling point set Q in the observation space is s Along the observation angle optical path d obs Perform projection transformation and map it into clothing mask image m g Projection mask UV on s , according to UV s The position encoding pair I g The feature map extracted by the 2D encoder is sampled by pixels and features to obtain the sampled pixel mapping feature F p and the sampled pixel color feature F c ; Finally, the sampled pixel map feature F is fused p and the sampled pixel color feature F c , get the clothing mapping feature F mapping ; The specific process of Step 3 is as follows: First, by analyzing the style feature F style , spatial feature F space and mapping feature F mapping Fusion is performed to generate the fusion feature F of clothing fus ; Then, the clothing feature decoding module based on neural radiation field is used to decode the fusion feature F fus Decode clothing features by defining a clothing feature decoding module D based on neural radiation field Nerf :(F fus ,q c ,d obs )→(c c ,σ c ), where q c and d obs is the query point and observation light path in the clothing specification space, and finally obtains the clothing query point color c in the specification space for neural radiation field rendering c With density σ c ; The specific process of Step 4 is as follows: First, the query point q on the clothing in the target posture space t By inverse transformation to a point in canonical space in, and normative space clothing point cloud Points in There is the closest Euclidean distance; Then define and In the specified observation optical path d t With the same color and density field function f Nerf ,Right now Specify the observation optical path d t The clothing query point q in the target pose space under t The color of the neural radiation field c t With density σ t It is expressed as follows: D Nerf (q t ,d t ,F style ,F Space ,F Mapping )→(c t ,σ t ); Finally, based on the color, transparency and rendering parameters of the neural radiation field, the spatial voxel integration is performed to calculate the two-dimensional image pixel I of the clothing neural rendering. r (r) as follows: where r(t) = o + td t It represents the direction from the camera optical center o along d t The reverse path ray in the direction, t u and t l is the upper and lower bounds of the propagation of r(t) in the clothing sampling area, and the volume density σ t (r(t)) represents the particle density of ray r(t) at point t, and the voxel color value c t (r(t),d t ) is represented by the ray r(t) along d t The color value at direction t, Where T(t) represents the ray from the lower bound t l The cumulative transmittance of the propagation process to t, I r (r) is the final clothing rendering result.

Images