A method for learning spatiotemporal features of dressed human bodies from monocular video based on neural radiation fields

Abstract

The present invention relates to a method for learning spatiotemporal features of dressed human bodies in monocular videos based on neural radiation fields, and belongs to the fields of computer vision and graphics and images. First, a time-correlated multilayer perceptron is used to perform temporal feature learning on monocular video frames in an input dressed human body data set to obtain temporal enhancement information of the dressed human body; then, a temporal-guided spatial feature learning module is established to perform point-level feature and pixel alignment feature learning to obtain global human body structure information and clothing local detail information respectively; then, self-attention is introduced to perform spatial feature fusion on global human body structure and clothing local details to obtain dressed human body fusion features; finally, by performing neural radiation field rendering on the dressed human body fusion features, the density value and color value of the reconstructed dressed human body are predicted. The predicted color and density can effectively generate a continuous and consistent new view of the dressed human body, and accurately reconstruct a dressed human body model with wrinkle details.

Description

Technical Field

[0001] The invention relates to a method for learning spatiotemporal features of a dressed human body in a monocular video based on a neural radiation field, and belongs to the field of computer vision and graphics. Background Art

[0002] With the commercial development of virtual digital humans and related industries, people have higher requirements for the accuracy of reconstructed models of clothed human bodies. The method based on explicit representation models the 3D clothing geometry on the human body through 3D point clouds or mesh templates, which can accurately restore the posture shape and clothing surface of the three-dimensional clothed human body, but cannot accurately represent the consistent and continuous moving human body, and it is difficult to learn the rich contextual semantic information between connected frames. The method based on implicit functions can accurately restore the surface details of clothing, but for input data with problems such as depth blur and insufficient posture observation, the reconstruction will still produce local floating artifacts. The method based on neural radiation field can process video streams faster, and at the same time can improve the generalization ability and rendering quality of the model. For example, Chen ( <eurographics>,2022) used a four-dimensional continuous time-varying function to simulate the representation of the dynamic human body in a static monocular camera, and learned the proposed representation by optimizing the dynamic scene, realizing the synthesis of new views of the dynamic human body from any viewpoint and at any time. Hu (<IEEE InternationalConference on Computer Vision> ,2023) utilizes 3D-aware hierarchical features to learn fine-grained and global features to restore texture details and supplement the missing information in some observations, thereby achieving the reconstruction of a generalizable human NeRF model from a single image. Yu(<Conference on Computer Vision and Pattern Recognition> ,2023) proposed a shared bidirectional deformation module to achieve generalizable front-end and back-end consistent deformation, and proposed a forward search correspondence module to query the corresponding appearance features to guide the rendering step.

[0003] Due to the existence of temporal information in monocular video, the information content of any frame in the motion sequence has a strong similarity with the information content of its adjacent frames, and the known methods fail to make full use of the contextual information between connected frames; for each individual frame in the motion sequence, there is global human structure information and local clothing texture information, which is crucial for reconstructing accurate geometric structure and color details, but the known methods are difficult to effectively learn and utilize the spatial information in the moving human body; the human body features in motion are scattered in the spatiotemporal domain, and it is difficult for the network to effectively fuse the spatiotemporal features of the human body in motion, resulting in the accumulation of geometric errors over time, so that the reconstructed results produce local shape loss and artifacts. The present invention will firstly learn the temporal features of the dressed human body to obtain continuous and consistent temporal enhancement information to supplement the posture-related motion of the modeled human in the video; guide the point-level features and pixel alignment features to learn spatial features through temporal features, global human structure information and local clothing detail information, and improve the accuracy of modeling the dressed human body and clothing surface details; construct a self-attention feature fusion module to fuse the point-level features and pixel alignment features to reduce the floating artifacts of the reconstruction results. Summary of the invention

[0004] The present invention provides a method for learning spatiotemporal features of a clothed human body from a monocular video based on a neural radiation field, which is used to solve the problems of feature blur and detail loss caused by monocular video reconstruction, and to more accurately reconstruct a clothed human body model with wrinkle details.

[0005] The technical solution of the present invention is: a method for learning the spatiotemporal characteristics of a dressed human body from a monocular video based on a neural radiation field, comprising the following steps:

[0006] Step 1: For the monocular video frames in the input dressed human body dataset, a time-correlated multi-layer perceptron is used to learn the temporal features to obtain consistent and continuous temporal enhancement information;

[0007] Step 2: Establish a time-series guided spatial feature learning module to learn point-level features and pixel alignment features, and obtain global human structure information and local clothing detail information respectively;

[0008] Step 3: Introduce self-attention to fuse the spatial features of the global human body structure and the local details of clothing to obtain the fusion features of the dressed human body;

[0009] Step 4: By performing neural radiation field rendering on the fusion features of the dressed human body, the density value and color value of the reconstructed dressed human body are predicted.

[0010] The specific process of Step 1 is as follows:

[0011] First, input the dressed human body dataset D = {D1, D2, ..., D n }Monocular video frame F t ∈D, at time t i The dressed human body is represented as a point x in the observation space o ;

[0012] Secondly, given time t i The three-dimensional coordinates of the dressed human body in the observation space are x o , using the position encoding function γ(x o ) and γ(t i ), the input three-dimensional coordinate x o and time step t are promoted to the high-dimensional space of neural radiation fields and motion fields;

[0013] Then, we transform M backward by predicting the time series b :(x o ,t i )→x c and temporal forward deformation M f :(x c ,t i )→x o' , define the same motion weight, and change x o Transform to the three-dimensional coordinate x in the canonical space c , and the three-dimensional coordinate x in the canonical space c Transform back to the observation space three-dimensional coordinate x o' ;

[0014] Finally, by using the three-dimensional coordinate x in the original observation space o And the transformed observation space three-dimensional coordinate x o' Define the front and back deformation consistency loss L consis =L2(x o ,M f (M b (x o ,t i ),t i )), where M b is the temporal backward deformation, M f It is the temporal forward deformation, and L2 represents the calculation of the Euclidean distance loss, which regularizes the error of the temporal deformation in the motion field over time, and finally obtains the temporal enhancement information of the dressed human body.

[0015] The specific process of Step 2 is as follows:

[0016] Step 2.1 Point-level feature learning

[0017] First, t i Time Frame View F t Perform two-dimensional encoding to obtain a two-dimensional feature map of the current frame view;

[0018] Secondly, from t i Extract SMPL vertices from the observation space at that time, and extract point-level features of the visible part by projecting the SMPL vertices onto the two-dimensional feature map of the current frame view;

[0019] Then, through the temporal backward deformation M b Guide the SMPL vertices to transform so that they are deformed to the canonical SMPL vertices, and voxelize the canonical SMPL vertices into a sparse three-dimensional volume tensor;

[0020] Finally, global point-level features are extracted from the encoded sparse 3D volume tensor, and the feature learning process is defined as: global =∏(W(F t ))+M b (SMPL,θ,β), where W represents the 2D encoder, П represents the 3D to 2D projection operator, and F t Yes i Time frame view, time series backward deformation M b Transform the SMPL vertices, posture θ and shape β to the canonical SMPL vertices in the canonical space, and finally obtain the global human structure information f of the dressed human body global .

[0021] Step 2.2 Pixel alignment feature learning

[0022] Start with t i Extract the canonical 3D coordinates x from the canonical space c , through the temporal forward deformation M f Guided norm point x c Transform it to the posture vertex of the observation space;

[0023] Then the posture vertices are projected to the current frame view and its two-dimensional feature map for grid sampling. The process of extracting local pixel alignment features is defined as follows: local =∏(W(F t ))+M f (x c ,θ,β), where the projection operation ∏(W(F t )) is consistent with the point-level features, and the temporal forward deformation M f The standard point x c Transform to the posture vertices of posture θ and shape β in the observation space to obtain the local detail information f of the clothing of the dressed person local .

[0024] The specific process of Step 3 is as follows:

[0025] First, the global human structure information f with temporal enhancement information is global and clothing details local After splicing, the convolution projection operation is used for preprocessing of feature fusion to obtain the convolution tensor Conv;

[0026] Then, the convolution tensor Conv is used as input to build a Transformer-based self-attention feature fusion module, and the structure of its self-attention module is changed to include the global human structure information f global and clothing details local The spatial features of the fusion feature F are obtained by feature fusion f , specifically expressed as where Attn(.) is the self-attention structure, the query Q, key K and value V are obtained by projecting hierarchical features using MLP, d is the scaling factor, and Softmax is used as the activation function.

[0027] The beneficial effects of the present invention are:

[0028] 1. The known method mainly uses a four-dimensional continuous time-varying function to simulate the representation of the dynamic human body in a static monocular camera. The structure is relatively simple, but the span of the motion semantic information between consecutive frames is large, resulting in the failure of the neural radiation field to fully utilize the context information between connected frames. The present invention defines a temporal backward deformation and a temporal forward deformation with the same motion weight in the motion field connected by the observation space and the normative space, and adds a forward and backward deformation consistency loss to perform temporal feature learning on the three-dimensional dressed human body in the monocular video, and optimizes the related motion postures of the dressed human body reconstruction through the similar information between the connected frames.

[0029] 2. The known methods mainly use three-dimensional perception hierarchical features to learn fine-grained and global features, but due to the lack of appropriate temporal information of the moving human body as a guide, the model is prone to overfitting or reconstruction deviation. The present invention uses a temporal deformation guidance network with temporal information to learn the spatial features of the dressed human body, and represents it from the global human body structure to the local clothing details, thereby improving the accuracy of reconstruction of the dressed human body and clothing surface details in motion.

[0030] 3. Most of the known methods adopt direct splicing or use neural networks for hierarchical feature fusion, which makes it difficult to effectively fuse the spatiotemporal features of the human body in motion. The present invention first performs convolution projection operations on the spatial features with temporal enhancement information, then constructs the Transformer self-attention module to fuse the features, and uses the fused features to guide the neural radiation field for reconstruction, which can ensure the consistency of the temporal sequence and reduce the floating artifacts of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 It is a timing characteristic flow chart of the present invention;

[0033] Figure 3 It is the point-level feature flow chart of the present invention;

[0034] Figure 4 It is a flowchart of the pixel alignment feature of the present invention;

[0035] Figure 5 It is a feature fusion flow chart of the present invention;

[0036] Figure 6 A flowchart for rendering reconstruction of the present invention;

[0037] Figure 7 This is an example diagram of the reconstruction result of a clothed human body according to the present invention. DETAILED DESCRIPTION

[0038] Example 1: Figure 1-Figure 7 As shown, a method for learning spatiotemporal features of dressed human bodies in monocular video based on neural radiation fields includes:

[0039] Step 1: For the monocular video frames in the input dressed human body dataset, a time-correlated multi-layer perceptron is used to learn the temporal features to obtain consistent and continuous temporal enhancement information;

[0040] Step 2: Establish a time-series guided spatial feature learning module to learn point-level features and pixel alignment features, and obtain global human structure information and local clothing detail information respectively;

[0041] Step 3: Introduce self-attention to fuse the spatial features of the global human body structure and the local details of clothing to obtain the fusion features of the dressed human body;

[0042] Step 4: By performing neural radiation field rendering on the fusion features of the dressed human body, the density value and color value of the reconstructed dressed human body are predicted.

[0043] The specific process of Step 1 is as follows:

[0044] First, input the dressed human body dataset D = {D1, D2, ..., D n }Monocular video frame F t ∈D, at time t i The dressed human body is represented as a point x in the observation space o ;

[0045] Secondly, given time t i The three-dimensional coordinates of the dressed human body in the observation space are x o , using the position encoding function γ(x o ) and γ(t i ), the input three-dimensional coordinate x o and time step t are promoted to the high-dimensional space of neural radiation fields and motion fields;

[0046] Then, we transform M backward by predicting the time series b :(x o ,t i )→x c and temporal forward deformation M f :(x c ,t i )→x o' , define the same motion weight, and change x o Transform to the three-dimensional coordinate x in the canonical space c , and the three-dimensional coordinate x in the canonical space c Transform back to the observation space three-dimensional coordinate x o' ;

[0047] Finally, by using the three-dimensional coordinate x in the original observation space o And the transformed observation space three-dimensional coordinate x o' Define the front and back deformation consistency loss L consis =L2(x o ,M f (M b (x o ,t i ),t i )), where M b is the temporal backward deformation, M f It is the temporal forward deformation, and L2 represents the calculation of the Euclidean distance loss, which regularizes the error of the temporal deformation in the motion field over time, and finally obtains the temporal enhancement information of the dressed human body.

[0048] The specific process of Step 2 is as follows:

[0049] Step 2.1 Point-level feature learning

[0050] First, t i Time Frame View F t Perform two-dimensional encoding to obtain a two-dimensional feature map of the current frame view;

[0051] Secondly, from t i Extract SMPL vertices from the observation space at that time, and extract point-level features of the visible part by projecting the SMPL vertices onto the two-dimensional feature map of the current frame view;

[0052] Then, through the temporal backward deformation M b Guide the SMPL vertices to transform so that they are deformed to the canonical SMPL vertices, and voxelize the canonical SMPL vertices into a sparse three-dimensional volume tensor;

[0053] Finally, global point-level features are extracted from the encoded sparse 3D volume tensor, and the feature learning process is defined as: global =Π(W(F t ))+M b (SMPL,θ,β), where W represents the 2D encoder, П represents the 3D to 2D projection operator, and F t Yes i Time frame view, time series backward deformation M b Transform the SMPL vertices, posture θ and shape β to the canonical SMPL vertices in the canonical space, and finally obtain the global human structure information f of the dressed human body global .

[0054] Step 2.2 Pixel alignment feature learning

[0055] Start with t i Extract the canonical 3D coordinates x from the canonical space c , through the temporal forward deformation M f Guided norm point x c Transform it to the posture vertex of the observation space;

[0056] Then the posture vertices are projected to the current frame view and its two-dimensional feature map for grid sampling. The process of extracting local pixel alignment features is defined as follows: local =Π(W(F t ))+M f (x c ,θ,β), where the projection operation Π(W(F t )) is consistent with the point-level features, and the temporal forward deformation M f The standard point x c Transform to the posture vertices of posture θ and shape β in the observation space to obtain the local detail information f of the clothing of the dressed person local .

[0057] The specific process of Step 3 is as follows:

[0058] First, the global human structure information f with temporal enhancement information is global and clothing details local After splicing, the convolution projection operation is used for preprocessing of feature fusion to obtain the convolution tensor Conv;

[0059] Then, the convolution tensor Conv is used as input to build a Transformer-based self-attention feature fusion module, and the structure of its self-attention module is changed to include the global human structure information f global and clothing details local The spatial features of the fusion feature F are obtained by feature fusion f , specifically expressed as where Attn(.) is the self-attention structure, the query Q, key K and value V are obtained by projecting hierarchical features using MLP, d is the scaling factor, and Softmax is used as the activation function.

[0060] Example 2: Figure 1 As shown, the specific steps of this method are as follows:

[0061] Step 1: First, learn the temporal features and input the dressed human body dataset D = {D1, D2, ..., D n }Monocular video frame F t ∈D, using the neural radiation field at time t i The dressed human body is represented as a point x in the observation space o , the point x in the observation space o The color and density model is represented by F r (M f (M b (x o ,t i ),F f )→c(x c ),σ(x c ), where M b and M f The time t i and the observation space x o The point in is used as the input of the temporal backward and temporal forward deformation mapping, and the output is the point x in the normative space c , F r is a point x in the canonical space c And the fusion feature F f is a mapping network that inputs and outputs its color value c and density value σ;

[0062] Secondly, given time t i The three-dimensional coordinates of the dressed human body in the observation space are x o , using the position encoding function γ(x o ) and γ(t i ), the input three-dimensional coordinate x o and time step t are promoted to the high-dimensional space of neural radiation field and motion field to learn high-frequency functions in the low-dimensional problem domain of neural radiation field;

[0063] Then, if Figure 2 As shown, in order to represent the moving objects in dynamic scenes, a time-dependent parameterized network MLP is introduced in the motion field to predict the temporal backward deformation M b :(x o ,t i )→x c and temporal forward deformation M f :(x c ,t i )→x o' , define the same motion deformation weight in the standard space, and the weight is independent of the posture, and the three-dimensional coordinates x of the dressed human body are o Transform to the three-dimensional coordinate x in the canonical space c , and the three-dimensional coordinate x in the canonical space c Transform back to the observation space three-dimensional coordinate x o' ;

[0064] Finally, by using the three-dimensional coordinate x in the original observation space o And the transformed observation space three-dimensional coordinate x o' Define the consistency loss L between the previous and next deformations that is only related to the deformation field consis =L2(x o ,M f (M b (x o ,t i ),t i )), where M b is the temporal backward deformation, M f It is the temporal forward deformation, L2 represents the calculation of the Euclidean distance loss, which regularizes the error of the temporal deformation in the motion field over time, and sets a threshold for L2 to avoid excessive regularization of the loss, and finally obtains the temporal enhancement information of the dressed human body.

[0065] Step 2.1, learn point-level features, such as Figure 3 As shown, the residual network ResNet18 is first used to t i Time Frame View F t Perform two-dimensional encoding to obtain a two-dimensional feature map of the current frame view;

[0066] Secondly, from t i The SMPL vertices are extracted from the observation space at that time, and the UV coordinates after projection are obtained by projecting the SMPL vertices to the two-dimensional feature map of the current frame view. The UV coordinates are then sampled using bilinear interpolation to extract the point-level features of the visible part.

[0067] Then, through the temporal backward deformation M b Guide the SMPL vertices extracted from the observation space to transform them to the canonical SMPL vertices, and voxelize the canonical SMPL vertices into a sparse three-dimensional volume tensor;

[0068] Finally, global point-level features are extracted from the encoded sparse 3D volume tensor, and the feature learning process is defined as: global =Π(W(F t ))+M b (SMPL,θ,β), where W represents the 2D encoder, П represents the 3D to 2D projection operator, and F t Yes i Time frame view, time series backward deformation M b Transform the SMPL vertices, posture θ and shape β to the canonical SMPL vertices in the canonical space, and finally obtain the global human structure information f of the dressed human body global .

[0069] Step 2.2, such as Figure 4 As shown, the pixel alignment feature is learned, starting from t i Extract the canonical 3D coordinates x from the canonical space c , through the temporal forward deformation M f Guide the specification of the 3D coordinate x c Transform it to the posture vertex of the observation space;

[0070] Then project the posture vertices to the current frame view to obtain the UV coordinates of the current frame view. The pixel features are obtained by bilinear interpolation sampling of the UV coordinates and the two-dimensional feature map of the frame view. The pixel colors are obtained by bilinear interpolation of the UV coordinates and the current frame view. The pixel features and pixel colors are then concatenated to obtain the pixel alignment features. The process of extracting local pixel alignment features is defined as follows: local =Π(W(F t ))+M f (x c ,θ,β), where the projection operation Π(W(F t )) is consistent with the point-level features, and the temporal forward deformation M f The standard point x c Transform to the posture vertices of posture θ and shape β in the observation space to obtain the local detail information f of the clothing of the dressed person local .

[0071] Step 3, if Figure 5 As shown in the figure, feature fusion is performed. First, the global human structure information f with temporal enhancement information is global and clothing details local After splicing, the spliced ​​features are preprocessed by convolution projection operation for feature fusion. The input feature dimension is 64 and the output dimension is 32, and the convolution tensor Conv is obtained.

[0072] Then, the convolution tensor Conv is used as input to construct a Transformer-based self-attention feature fusion module, and the dimensional structure of its self-attention module is changed to include the global human structure information f global and clothing details local The spatial features of the fusion feature F are obtained by feature fusion f , specifically expressed as where Attn(.) is the self-attention structure, the query Q, key K and value V are obtained by projecting hierarchical features using MLP, d is the scaling factor, and Softmax is used as the activation function.

[0073] Step 4, such as Figure 6 As shown, in the fusion feature F f Under the guidance of c And the fusion feature F f As input, and output its color c(x c ) and density σ(x c ), the entire neural rendering network is represented as a mapping F r :c(x c ),σ(x c )=F r (γ(x c ),F f ) where γ is x c The sinusoidal position encoding is finally used to draw the dressed human body model in the canonical space using the Nerf body rendering equation.

[0074] The results of reconstructing the clothed human body from the video frame are as follows: Figure 7 As shown, Figure 7 The left side shows that the reconstructed model has a high accuracy through four viewing angles. Figure 7 The reconstruction result in the right motion shows that the present invention has a better reconstruction effect, and the reconstruction result has almost no artifacts.

[0075] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.< / eurographics>

Claims

1. A method for learning spatiotemporal features of human bodies in monocular video based on neural radiation fields, characterized by: The steps include: Step 1: For the monocular video frames in the input dressed human body dataset, a time-correlated multi-layer perceptron is used to learn the temporal features to obtain consistent and continuous temporal enhancement information; Step 2: Establish a time-series guided spatial feature learning module to learn point-level features and pixel alignment features, and obtain global human structure information and local clothing detail information respectively; Step 3: Introduce self-attention to fuse the spatial features of the global human body structure and the local details of clothing to obtain the fusion features of the dressed human body; Step 4: By performing neural radiation field rendering on the fusion features of the dressed human body, the density value and color value of the reconstructed dressed human body are predicted; The specific process of Step 1 is as follows: First, input the dressed human body dataset D = {D1, D2, ..., D n }Monocular video frame F t ∈D, at time t i The dressed human body is represented as a point x in the observation space o ; observation space midpoint x o The color and density model is represented by F r (M f (M b (x o ,t i ),F f )→c(x c ),σ(x c ), where M b and M f is the time t i and the observation space x o The point in is used as the input of the temporal backward and temporal forward deformation mapping, and the output is the point x in the normative space c , F r is a point x in the canonical space c And the fusion feature F f is a mapping network that inputs and outputs its color value c and density value σ; Secondly, given time t i The three-dimensional coordinates of the dressed human body in the observation space are x o , using the position encoding function γ(x o ) and γ(t i ), the input three-dimensional coordinate x o and time step t are promoted to the high-dimensional space of neural radiation fields and motion fields; Then, we transform M backward by predicting the time series b :(x o ,t i )→x c and temporal forward deformation M f :(x c ,t i )→x o' , define the same motion weight, and change x o Transform to the three-dimensional coordinate x in the canonical space c , and the three-dimensional coordinate x in the canonical space c Transform back to the observation space three-dimensional coordinate x o' ; Finally, by using the three-dimensional coordinate x in the original observation space o And the transformed observation space three-dimensional coordinate x o' Define the front and back deformation consistency loss L consis =L2(x o ,M f (M b (x o ,t i ),t i )), where M b is the temporal backward deformation, M f It is the temporal forward deformation, L2 represents the calculation of the Euclidean distance loss, which regularizes the error of the temporal deformation in the motion field over time, and finally obtains the temporal enhancement information of the dressed human body; The temporal guided spatial feature learning module in Step 2 includes the following: Step 2.1 Point-level feature learning; First, t i Time Frame View F t Perform two-dimensional encoding to obtain a two-dimensional feature map of the current frame view; Secondly, from t i Extract SMPL vertices from the observation space at that time, and extract point-level features of the visible part by projecting the SMPL vertices onto the two-dimensional feature map of the current frame view; Then, through the temporal backward deformation M b Guide the SMPL vertices to transform so that they are deformed to the canonical SMPL vertices, and voxelize the canonical SMPL vertices into a sparse three-dimensional volume tensor; Finally, global point-level features are extracted from the encoded sparse 3D volume tensor, and the feature learning process is defined as: global =Π(W(F t ))+M b (SMPL,θ,β), where W represents the 2D encoder, П represents the 3D to 2D projection operator, and F t Yes i Time frame view, time series backward deformation M b Transform the SMPL vertices, posture θ and shape β to the canonical SMPL vertices in the canonical space, and finally obtain the global human structure information f of the dressed human body global ; The temporal guided spatial feature learning module in Step 2 also includes the following: Step 2.2 Pixel alignment feature learning Start with t i Extract the canonical 3D coordinates x from the canonical space c , through the temporal forward deformation M f Guided norm point x c Transform it to the posture vertex of the observation space; Then the posture vertices are projected to the current frame view and its two-dimensional feature map for grid sampling. The process of extracting local pixel alignment features is defined as follows: local =Π(W(F t ))+M f (x c ,θ,β), where the projection operation Π(W(F t )) is consistent with the point-level features, and the temporal forward deformation M f The standard point x c Transform to the posture vertices of posture θ and shape β in the observation space to obtain the local detail information f of the clothing of the dressed person local ; The specific process of Step 3 is as follows: First, the global human structure information f with temporal enhancement information global and clothing details local After splicing, the convolution projection operation is used for preprocessing of feature fusion to obtain the convolution tensor Conv; Then, the convolution tensor Conv is used as input to construct a Transformer-based self-attention feature fusion module, and the structure of its self-attention module is changed to include the global human structure information f global and clothing details local The spatial features of the fusion feature F are obtained by feature fusion f , specifically expressed as Where Attn(.) is the self-attention structure, the query Q, key K and value V are obtained by projecting hierarchical features using MLP, d is the scaling factor, and Softmax is used as the activation function; Step 4 includes: In the fusion feature F f Under the guidance of c And the fusion feature F f As input, and output its color c(x c ) and density σ(x c ), the entire neural rendering network is represented as a mapping F r :c(x c ),σ(x c )=F r (γ(x c ),F f ), where γ is x c The sinusoidal position encoding is finally used to draw the dressed human body model in the canonical space using the Nerf body rendering equation.

Images