Monocular video garment texture replacement method based on grid texture

Abstract

The application discloses a monocular video clothing texture replacement method based on grid texture, which proposes a hybrid parameter multilayer perceptron to obtain the UV of the clothing part in the video pixel by pixel, and in order to correctly reflect the wrinkles, folding and occlusion of the clothing and reduce the distortion effect of the texture, a UV gradient constraint is adopted in the UV regression process. In order to improve the continuity of the UV in the time sequence change and reduce the flicker after the final texture replacement, the method uses a time sequence constraint, uses the UV of the similar frames to constrain the UV of the current frame. The method also extracts a shadow layer from the clothing part of the video to reflect the real light and shadow effect. The method is suitable for various styles of clothing, various types of environmental light and challenging human actions. The method is easy to use, has low requirements on equipment, and can produce high-quality results. In the fashion, e-commerce and advertising fields, the method has rich use prospects.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video editing and image synthesis, and in particular to a monocular video garment texture replacement method based on grid texture. BACKGROUND

[0002] The rapidly developing advertising and e-commerce industry today has brought about a huge market, and the growing market has made the demand for online clothing display by practitioners more intense. Online clothing display often faces the scenario of combining the same cut style of clothing with different patterns, so that the merchant can choose multiple attempts of patterns and select the best appearance to present to the customer. However, the current workflow usually requires the same garment to be made with different patterned fabrics and tried on by a model, or relies on manual editing of the captured clothing video, which is time-consuming and laborious.

[0003] The mainstream method to realize human appearance replacement today is to estimate the pixel-by-pixel UV for the human body part in the picture or video, and combine the given texture to realize automatic appearance replacement. For example, some researchers use optical flow to divide the scene in the video into multiple levels of foreground and background, and assign independent UV to each level. Or, train the network and infer to find the UV correspondence for each pixel of the human body in the video. Some researchers estimate UV independently for each part of the body, such as the head, body, and arms, making the appearance editing more precise. Others extend the UV estimation of the human body in the video to the clothes, making the appearance editing transferable to loose clothing such as skirts.

[0004] However, these methods are not suitable for clothing with complex structures, and the resulting quality is not sufficient to support subsequent applications. And this kind of method is only suitable for the case where the clothing is close to the human body, and is not suitable for loose clothing such as skirts. Although some researchers obtain the normal vector information from the human body surface to restore the UV that conforms to the wrinkle characteristics of the clothing surface, producing a realistic texture replacement effect. However, this method cannot be used on clothing with self-occlusion and folding, and cannot reflect the overlapping effect of loose skirts.

[0005] Another type of appearance replacement method is based on a given texture. For example, a color triangle texture is used to make a garment, and the garment geometry is reconstructed, and the geometry appearance is further edited. Alternatively, a color dot texture is used, and the garment geometry is reconstructed using multi-view. Alternatively, a color grid texture is used, and the garment geometry is reconstructed using high-precision multi-view cameras, and the appearance editing is achieved. However, the above methods are mainly designed for 3D garment reconstruction, and are not suitable for garment appearance editing based on a single video. This is because the reconstructed garment is still difficult to accurately capture the wrinkle details of the garment, and most users cannot afford the cost of using several synchronous high-precision cameras for shooting in a photo studio. Further, the above methods are also difficult to handle arbitrary shooting environments, motion blur and complex human motions, limiting their application in practice. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a monocular video garment texture replacement method based on a grid texture. A model wears a garment printed with a specific grid texture, and a cameraman shoots a motion video of the model wearing the garment using a general recording device (such as a camera or a mobile phone). The method of the present application can automatically recover the UV of the grid texture garment in the video. Combined with the extracted shadow and the given texture picture, the video garment can be replaced with other realistic texture appearance. The present application has low requirements for equipment, is suitable for various lighting environments, is not sensitive to motion blur, and is suitable for models to make various challenging motions. The present application can be applied to the e-commerce and advertising industries, and employees can conveniently adjust the appearance of the model to present the best display effect.

[0007] The object of the present application is achieved by the following technical solutions:

[0008] A monocular video garment texture replacement method based on a grid texture, comprising the following steps:

[0009] Step 1: simulate a piece of grid texture fabric using a physical simulator, and use a renderer to render the result as synthetic training data;

[0010] Step two: construct and train the grid network, color network, shadow network and UV gradient network based on the U-Net architecture, the input of the grid network, color network and shadow network is the original image of HxWx3, the output of the grid network is the image of HxWx3, each pixel on the image contains three channels, which respectively represent the probability of the pixel being a grid center point, the probability of the pixel being a grid corner point and the probability of the pixel being other; the output of the color network is the image of HxWx8, each pixel on the image contains eight channels, which respectively represent the probability of the pixel being eight colors of grid texture; among them, the eight colors include red, blue, yellow, green, cyan, purple, white and black, black is the boundary color of the grid, and the rest is the color of the grid itself; the output of the shadow network is the image of HxWx1, that is, the shadow layer; the input of the UV gradient network is the original image of HxWx3 and the UV gradient direction of HxWx4, and the output is the UV gradient of HxWx4;

[0011] Step three: using the same grid texture of the cloth in step one, cut the clothes and wear them by the model, shoot the model action to generate a video;

[0012] Step four: input each frame of the video in step three into the trained grid network, color network, shadow network, UV gradient network and optical flow prediction network respectively, extract the corner point, center point and color category information of the image of the garment part, the shadow layer and the UV gradient of each frame of the video, and the forward and backward optical flow of the adjacent two frames of the video; and based on the corner point, center point and color category information of each frame of the image of the garment part, the corresponding point detection algorithm is used to identify the UV corresponding point;

[0013] Step five: construct and train a hybrid parameter multilayer perceptron, the hybrid parameter multilayer perceptron includes a dimension lifting layer based on random Fourier features, and four hidden layers with a size of 256, the hybrid parameter multilayer perceptron defines a set of MLP parameters for each key frame of the video, and the MLP parameters change continuously with time t; the input of the hybrid parameter multilayer perceptron is the serial number of each frame and the pixel coordinates of the image of the frame, and the output is the UV coordinates; when training the hybrid parameter multilayer perceptron, the UV corresponding point constraint and the time sequence constraint are set;

[0014] Step six: input each frame of the video obtained in step three into the trained hybrid parameter multilayer perceptron to obtain the UV coordinates of each pixel of each frame and the UV gradient direction; input the UV gradient direction and each frame of the video into the trained UV gradient network to extract the UV gradient of the garment part frame by frame; based on the UV gradient, the hybrid parameter multilayer perceptron is trained again, and the UV gradient constraint is additionally applied to obtain the hybrid parameter multilayer perceptron after secondary training;

[0015] Step seven: obtain the UV coordinates of each pixel in each frame by the mixed parameter multilayer perception machine after secondary training, and combine the shadow layer of the clothing part obtained in step three to obtain the clothing texture replacement result with real shadow effect.

[0016] Further, the step one comprises the following sub-steps:

[0017] (1.1) Construct a piece of cloth in the simulator, place it on an irregular geometric body, drive the irregular geometric body to move, drive the cloth to move, and obtain a cloth simulation sequence;

[0018] (1.2) Attach a grid texture to the cloth simulation sequence in the renderer, and render to obtain an original map, a shadow layer, a grid corner and a center point, a grid color category, and a UV coordinate.

[0019] Further, the obtained UV coordinates are subjected to Sobel convolution kernel to obtain UV gradient; and the size of the UV gradient is normalized to obtain the UV gradient direction.

[0020] Further, in the step four, based on the corner point, center point and color category information of the clothing part of each frame image, a corresponding point detection algorithm is used to identify the UV corresponding point, which comprises the following sub-steps:

[0021] (1) Obtain a heterogeneous graph by connecting the corner point and the center point, and obtain an isomorphic graph by connecting the center point and the center point;

[0022] (2) Extract 3x3 grid color in the isomorphic graph to obtain the UV corresponding point;

[0023] (3) Use a plurality of grid texture voting algorithms to determine the grid texture of each UV corresponding point; and use a UV corresponding point growth algorithm to further search for more UV corresponding points from the boundary of the isomorphic graph.

[0024] Further, in the step five, for a video with a total frame number of T, every s frame is a key frame, and a total of MLP parameters are defined. θ i represents the i-th MLP parameter, which is defined at time i*s; for the t-th frame, the mixed parameter multilayer perception machine parameter is defined as a linear combination of the parameter set The mixed parameter multilayer perception machine has the following form:

[0025]

[0026] Wherein, alpha is a linear combination parameter, which is calculated by cubic spline interpolation; x is a pixel coordinate, and f is a multilayer perception machine network.

[0027] Further, when training the hybrid parameter multilayer perceptron, the UV corresponding point constraint L is set data is:

[0028]

[0029] wherein, is the pixel coordinate and UV of the i-th corresponding point of the t-th frame.

[0030] Further, the timing constraint includes a repair constraint and a continuity constraint; the UV coordinates of the isomorphic graph coverage are taken as accurate UV coordinates, and the UV coordinates not covered are taken as inaccurate UV coordinates, and the repair constraint is specifically to extract the accurate UV coordinates in the adjacent frame to repair the inaccurate UV coordinates of the current frame.

[0031] A monocular video clothing texture replacement system based on mesh texture, the system comprises one or more processors for realizing a monocular video clothing texture replacement method based on mesh texture.

[0032] An electronic device comprising:

[0033] One or more processors;

[0034] Storage device for storing one or more programs, when the one or more programs are executed by the electronic device, the electronic device realizes a monocular video clothing texture replacement method based on mesh texture.

[0035] A computer readable storage medium having a program stored thereon, the program being executed by a processor to realize a monocular video clothing texture replacement method based on mesh texture.

[0036] The beneficial effects of the present application are as follows:

[0037] (1) The present application proposes a hybrid parameter multilayer perceptron, and uses it to regress UV corresponding points to obtain pixel-by-pixel UV of the clothing part in the video. In order to make the UV obtained by regression correctly reflect the wrinkles, folds and occlusions of the clothing, and at the same time reduce the distortion effect of the texture, the method adopts a novel UV gradient constraint in the above UV regression process. Further, in order to improve the continuity of UV in the time sequence and reduce the flicker after the final texture replacement, the method uses a timing constraint to use the UV of the similar frame to constrain the UV of the current frame. The method also extracts a shadow layer from the clothing part of the video to reflect the real light and shadow effect. Finally, using the pixel-by-pixel UV and shadow layer obtained by the above steps, combined with the replacement texture given by the user, the user can replace the new texture with realistic texture for the clothing part in the video. The method is suitable for various styles of clothing, various environmental lighting and challenging human actions.

[0038] (2) The present application is easy to use, and requires low equipment. For any new model, the model only needs to wear the above-mentioned special grid texture clothing, and is directly shot by a cameraman holding a device, without the need for shooting in a multi-camera environment such as a photo studio. The present application has robustness and is suitable for various environmental lighting. The method is also applicable when the model makes difficult movements such as jumping, dancing, rotating, squatting and standing, and produces high-quality results. Compared with manually editing a video to obtain the result of replacing the appearance of clothing, the method can greatly reduce the labor cost of processing the video frame by frame.

[0039] (3) The method first proposes a monocular video high-quality clothing texture replacement method based on a given texture, which has rich use prospects in the fields of fashion, e-commerce and advertising. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a frame of image in a video shot by a monocular camera.

[0041] Figure 2 is the pixel-by-pixel UV of the clothing part obtained by using the method of the embodiment of the present application.

[0042] Figure 3 is the shadow layer of the clothing part extracted from a frame of image by the present application.

[0043] Figure 4 is the effect picture of replacing the grid texture clothing part with a new texture. DETAILED DESCRIPTION

[0044] The present application will be described in detail below according to the drawings and preferred embodiments, and the purposes and effects of the present application will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0045] The core of the present application is to extract key information (such as UV corresponding points, UV gradient, optical flow, etc.) from the clothing in the shot video, and use these information to assist a blended-weight multilayer perceptron (blended-weight MLP) regression to obtain the frame-by-frame UV of the video clothing. The method mainly includes seven steps: training data synthesis, network training, clothing making and data collection, video information extraction, video texture coordinate rough regression, video texture coordinate accurate regression, and video texture replacement. Among them, the video texture coordinate regression is the core step, and the training data synthesis, network training, clothing making and data collection, and video information extraction are auxiliary steps for key information extraction.

[0046] Firstly, the present application relies on a specific color grid texture. The grid texture contains eight colors, in which red, blue, yellow, green, cyan, purple, white are the grid colors, and black is the boundary color of the grid. In a generated grid texture, any 3x3 grid is unique after rotation, so the unique UV corresponding point can be obtained by identifying the 3x3 grid. The present application uses a resolution setting of 15mm per grid to adapt to ordinary monocular shooting conditions. Based on the grid texture, the steps of the present application are as follows.

[0047] Step one: training data synthesis; including the following sub-steps:

[0048] (1.1) cloth simulation. In the simulation software, a piece of cloth is placed on an irregular geometry, and the geometry is driven to make rotation, sudden stop and other actions. The cloth shakes with the geometry, generating wrinkles and occlusion and other cloth features. This way produces 500 frames of simulation sequences.

[0049] (1.2) cloth rendering. In the rendering software, the simulation sequence is pasted with the grid texture. When rendering each picture, the rendering software randomly selects the environment light and randomly selects the rendering angle for rendering. This way produces 500 frames of rendering results. For each result, the rendering can get the rendering original, the shadow layer, the UV coordinates, the grid corner and center points, and the grid color category.

[0050] Step two: build and train the grid point network, color network, shadow network and UV gradient network based on the U-Net architecture, for extracting key information from the video.

[0051] In this embodiment, 30K small pictures are obtained as training data by randomly cropping from the rendering data of step one. The following will introduce each network respectively.

[0052] (2.1) grid point network

[0053] The grid network uses U-Net as the network architecture. The network architecture from input to output is convolutional layer, max-pooling layer, convolutional layer, max-pooling layer, convolutional layer, max-pooling layer, convolutional layer, up-sampling layer, convolutional layer, up-sampling layer, convolutional layer, up-sampling layer, convolutional layer. Each convolutional layer contains two two-dimensional convolution operations. The two-dimensional convolution operation uses a 3x3 size convolution kernel, and uses batch normalization (Batch normalization) for normalization, and ReLU as the activation function. The max-pooling layer uses a 2x2 size convolution kernel, and reduces the input length and width size by half. The up-sampling layer uses bilinear interpolation, and increases the input length and width size by one time. The input of the grid network is an original image with a size of HxWx3, and the output is an image with a size of HxWx3. Each pixel on the output image contains three channels, representing the probability that the pixel is a grid center point, the probability that the pixel is a grid corner point, and the probability that the pixel is other. All the center points and corner points in the original image are obtained by the output probabilities.

[0054] (2.2) Color network

[0055] The color network uses the same network structure as the grid network, but the input is an original image with a size of HxWx3, and the output is an image with a size of HxWx8. Each pixel on the output image contains eight channels, representing the probability that the pixel is one of the eight colors of the grid texture. The color category of each pixel in the original image is obtained by the output probability.

[0056] (2.3) Shadow network

[0057] The shadow network uses the same network structure as the grid network, but the difference is that the input is an original image with a size of HxWx3, and the output is an image with a size of HxWx1, i.e. a shadow layer.

[0058] (2.4) UV gradient network

[0059] First, a Sobel convolution kernel is applied to the UV coordinates in the training data to obtain the UV gradient. Then the size of the UV gradient is normalized to obtain the UV gradient direction. Finally, the training is performed. The UV gradient network uses the same network structure as the grid network, but the difference is that the input is an original image with a size of HxWx3 and HxWx4 UV gradient direction, and the output is a UV gradient with a size of HxWx4.

[0060] In order to make the network suitable for real data, this step performs data enhancement before the above training, i.e. rotation, noise, blur, contrast brightness and saturation adjustment and other data enhancement strategies are performed on the original image, so that the network can still obtain accurate results when facing real data with more noise.

[0061] Step three: using the same grid texture fabric as step one, cutting the clothes and wearing them by the model, shooting the model's action to generate a video.

[0062] In this embodiment, using the same grid texture of the training data in step one, several 100x100 grid textures are made and printed on 1.5mx1.5m cloth. The tailor cuts the cloth to make various fashion garments. The model wears the garment and makes any action. The photographer holds a single-lens camera to shoot the model video, and the video resolution is 1080p.

[0063] Step four: video information extraction.

[0064] Each frame of the video in step three is input into the trained grid point network, color network, shadow network, UV gradient network and optical flow prediction network respectively, and the corner point, center point and color category information of the garment part image, the shadow layer, UV gradient, and the forward and backward optical flow of the adjacent two frames of the video are extracted frame by frame; and based on the corner point, center point and color category information of the garment part of each frame image, the corresponding point detection algorithm is used to identify the UV corresponding point.

[0065] But before extracting information from the video, first use SAM (code, https: / / github.com / facebookresearch / segment-anything) to segment the garment part frame by frame.

[0066] (4.1) UV corresponding point

[0067] The UV corresponding point refers to the corresponding relationship between the pixel position in the picture and the texture coordinate (UV). After extracting the corner point, center point and color category information of the garment part image in each frame of the video using the trained grid point network and color network, this embodiment uses a corresponding point detection algorithm to obtain the UV corresponding point, which includes the following sub-steps:

[0068] (4.1.1) Connect the corner point and the center point to obtain a heterogeneous graph, and connect the center point and the center point to obtain an isomorphic graph;

[0069] (4.1.2) Extract 3x3 grid color in the isomorphic graph to obtain the UV corresponding point.

[0070] After obtaining the UV corresponding point, the present application further uses the following steps to obtain the grid texture of the UV corresponding point and increase the number of corresponding points, as follows:

[0071] (4.1.3) Adopting the multi-grid texture voting algorithm to determine the grid texture of each UV corresponding point. This algorithm can further distinguish which grid texture each UV corresponding point belongs to. Specifically, for a corresponding point A, its 8 adjacent corresponding points vote for point A to determine which grid texture A belongs to. For the voting process, the adjacent corresponding point obtains its 3x3 grid around it and checks whether the 3x3 grid appears in each grid texture. If it appears in a certain grid texture, the adjacent corresponding point calculates the UV of point A and votes the UV to point A with the grid texture. If the repeated UV with the most occurrences in the votes obtained by point A is more than or equal to 3, it is considered that the UV wins in the votes, and the grid texture to which the UV belongs is assigned to A. Otherwise, A is deleted from the corresponding point set.

[0072] (4.1.4) Adopting the UV corresponding point growth algorithm to further search for more UV corresponding points from the isomorphic graph boundary. Specifically, if there is a 3x3 grid around the corresponding point B, and the grid is at the isomorphic graph boundary, then the 3x3 grid is translated outward by a single grid distance to expand and obtain a new 3x3 grid. After expansion, three new grid center points and their colors are obtained which are not in the isomorphic graph. If the new 3x3 grid appears in the grid texture to which the corresponding point B belongs, then the three new grid center points are assigned to the UV as new corresponding points and added to the isomorphic graph, and the above algorithm is repeated from the new 3x3 grid. If there is no 3x3 grid around the corresponding point B or the 3x3 grid around B is not at the isomorphic graph boundary, or the new 3x3 grid does not appear in the grid texture to which the corresponding point B belongs, then no processing is performed on the corresponding point B.

[0073] (4.2) Shadow layer

[0074] Input each frame of the video in step three into the shadow network to extract the shadow layer for the clothing part of each frame of the video.

[0075] (4.3) Optical flow

[0076] Input each frame of the video in step three into the optical flow prediction network to extract forward and backward optical flow for adjacent two frames of the video. The optical flow prediction network adopts the Raft algorithm in the prior art, the website is: https: / / github.com / princeton-vl / RAFT.

[0077] Step five: building and training a hybrid parameter multilayer perceptron

[0078] The purpose of this step is to establish a pixel-by-pixel inaccurate UV for each frame of the video as an input provided to the UV gradient grid. In order to achieve this goal, this step mainly uses a hybrid parameter multilayer perceptron to regress the UV corresponding point described in step 4.1 and applies constraints to improve the regression quality.

[0079] (5.1) Design of hybrid parameter multi-layer perceptron

[0080] The purpose of the design of hybrid parameter multi-layer perceptron is to reduce the jitter of the final problem replacement video result. The idea of this design is to take the network parameters of the multi-layer perceptron MLP as a function of time t, and the network parameters change continuously with time t. The specific description is as follows. Let the multi-layer perceptron MLP be defined as:

[0081] f(x; θ)

[0082] where x is the pixel coordinate, θ is the MLP network parameter, f is the MLP network, and the network output is UV. The dimension lifting layer based on random Fourier features, that is, the input is transformed based on multiple trigonometric functions as the basis, and the input is converted from a dimension of 2 to 256. Here, the random Fourier feature lifting network is used to learn the ability.

[0083] When the video texture coordinate is roughly regressed, the regression problem is described as follows. The total number of training videos is T frames, and the frame sequence number t is in the range of 0 to T. Define every s frames of a video as a key frame, and define a set of MLP parameters for the key frames of a video θ i represents the i-th MLP parameter, which is defined at time i*s. For a video with a total of T frames, a total of MLP parameters are defined. For the t-th frame, the parameter of the hybrid parameter multi-layer perceptron is defined as a linear combination of the parameter set The hybrid parameter multi-layer perceptron model is defined as has the following form:

[0084]

[0085] where t represents the sequence number of each frame, is the hybrid parameter multi-layer perceptron model, x represents the pixel coordinate, and a is the linear combination parameter. The model uses cubic spline interpolation to calculate a, so that the MLP parameters can change smoothly as time t changes. In actual operation, it is appropriate to set s=10 for a 30fps video.

[0086] The hybrid parameter multi-layer perceptron includes a dimension lifting layer based on random Fourier features and four hidden layers with a size of 256. The hybrid parameter multi-layer perceptron defines a set of MLP parameters for each key frame of the video, and the MLP parameters change continuously with time t; the input of the hybrid parameter multi-layer perceptron is the sequence number of each frame and the pixel coordinate of the image of the frame, and the output is the UV coordinate; when training the hybrid parameter multi-layer perceptron, the UV corresponding point constraint and the time sequence constraint are set.

[0087] (5.2) UV corresponding point constraint

[0088] This constraint connects the discrete UV correspondences by Regression to get per-pixel UV. If the UV correspondences from step 4.1 are where are the pixel coordinates and UV of the ith correspondence in the t-th frame, the correspondence constraint can be written as:

[0089]

[0090] (5.3) Temporal constraint

[0091] This constraint uses the optical flow from step 4.3 to constrain the UVs temporally so that the final result is continuous. This step first uses the optical flow to establish the connection between the pixels in adjacent frames. If the forward optical flow does not match the backward one, the corresponding pixel is filtered out. After the pixels are connected by the optical flow, two temporal constraints are applied to the model: the inpainting constraint and the consistency constraint.

[0092] For the inpainting constraint, it mainly takes the accurate UV part in the adjacent frame to repair the inaccurate UV part in the current frame. Specifically, the isomorphic map coverage part in 4.1.4 is taken as the accurate UV part, and the uncovered part is taken as the inaccurate UV part. In the accurate UV part, all the pixel UVs of the accurate UV part can be obtained by bilinear interpolation. For each pixel in the inaccurate UV part of the current frame, the accurate UV part in the adjacent frame is searched forward and backward, and its UV is supplemented to the current frame, denoted as t (x). The above supplemented UV is used to constrain the model The inpainting constraint can be obtained as:

[0093]

[0094] For the consistency constraint, this constraint directly constrains the pixels in adjacent frames. The consistency constraint is defined as

[0095]

[0096] where S t is the pixel of the grid texture clothing part in the t-th frame, t' e {t±1, t±2, t±3}, and x' is the pixel of x propagated from frame t to frame t' by the optical flow.

[0097] Combining the above constraints, the temporal constraint can be defined as,

[0098] L temp = L inpaint + L consist

[0099] Combining all the constraints, for The regression process, in which the final constraint is implemented, is...

[0100] L video =L data +λ temp L temp

[0101] Where λ temp This is a parameter for balancing the loss term. This step sets λ. temp =0.7. During training, for each type of constraint, this step randomly selects 10K pixels from all frames as a batch. This step trains for each video for 90K iterations, with the learning rate starting from 10. -3 Gradually decreased to 10 -5 .

[0102] Step 6: Input each frame of the video obtained in Step 3 into the trained hybrid parameter multilayer perceptron to obtain the UV coordinates of each pixel in each frame and the UV gradient direction; input the UV gradient direction and each frame of the video into the trained UV gradient network to extract the UV gradient of the clothing part frame by frame; based on the UV gradient, train the hybrid parameter multilayer perceptron again and apply additional UV gradient constraints to obtain the hybrid parameter multilayer perceptron after secondary training.

[0103] In this process, each frame of the video obtained in step three is input into the trained hybrid parameter multilayer perceptron to obtain the UV coordinates of each pixel in each frame and the UV gradient direction, thus obtaining pixel-wise coarse UV. The hybrid parameter multilayer perceptron is then trained again to obtain pixel-wise precise texture coordinates.

[0104] Specifically, obtaining pixel-by-pixel accurate texture coordinates includes:

[0105] (6.1) UV gradient information extraction

[0106] This method is based on pixel-wise coarse UV. A Sobel convolution kernel is applied to each pixel of the UV to obtain the UV gradient, and the magnitude of the UV gradient is normalized to obtain the UV gradient direction. The UV gradient direction and each frame of the video are input into the trained UV gradient network to extract the UV gradient of the clothing portion frame by frame.

[0107] (6.2) Precise Regression of Video Texture Coordinates

[0108] This step is used to retrain the hybrid parameter multilayer perceptron, but with the addition of UV gradient constraints. The UV gradient constraints constrain the UV gradients and, combined with L... 0.5 The loss function can produce the effect of UV transitions and reduce the distortion of the final result. This constraint uses the UV gradient information extracted in step 6.1 as supervision and is constructed as follows:

[0109]

[0110] Θ is the region where UV correspondence is missing. This constraint only constrains the region where there is no UV correspondence supervision. F is the Forbenius norm. is the Jacobian of the output UV with respect to the input x.9: R 2 → R 2×2 is the gradient of the UV. This constraint does not employ the common LI or L2 loss, but employs the L 0.5 loss. L 0.5 is defined as:

[0111] L 0.5 (x) = (x 2 + c 4 ) 0.25 - c

[0112] where c = 0.1 is used to control the loss function to be continuously differentiable at x close to 0.

[0113] Combining all the constraints, for the regression process of , the final implemented constraint is,

[0114] L video = L data + λ grad L grad + λ temp L temp

[0115] where λ grad = 0.005 is the parameter to balance the loss terms. Again, the hybrid parameter multilayer perceptron from the above step is used to perform pixel-wise inference to get the pixel-wise accurate UV.

[0116] Step Seven: Video Texture Replacement

[0117] With the UV coordinates of each pixel of each frame obtained by the hybrid parameter multilayer perceptron after the secondary training, and the shadow layer of the clothing part extracted frame by frame obtained in step three, the clothing texture replacement result with real shadow effect is obtained.

[0118] In this step, the pixel-wise accurate UV is obtained, and the albedo can be obtained by combining the given texture provided by the user. The albedo multiplied by the shadow layer can obtain the clothing texture replacement result with real shadow effect.

[0119] Implementation Example

[0120] ​In this embodiment, the key information extraction networks, including the grid network, color network, shadow network, UV gradient network and optical flow prediction network, are first trained. This network training step only needs to be performed once, and after that, there is no need to repeat the training for any video. On a computer configured with RTX3060ti, the training time of each U-Net network is 10 hours.

[0121] In the data collection stage, the inventors use a smartphone to shoot videos at a setting of 1080p, 30fps. As shown in FIG. 1, it is a frame of image in the video shot by a monocular camera. The video is processed on a desktop computer equipped with RTX4090. A single 300-frame video requires about half an hour for key information extraction and about 100 minutes for training to obtain pixel-wise UV. Figure 1 Figure 2 The pixel-wise UV of the clothing part obtained by using the method of the embodiment of the present application.

[0122] The inventors invite different models to wear different styles of grid texture clothing, make various actions in various environments to shoot videos, and generate texture replacement videos. Figure 3 is the shadow layer of the clothing part extracted from a frame of image by the present application, Figure 4 is the rendering image of the grid texture clothing part replaced with new texture. From these figures, it can be seen that the method of this embodiment can produce texture replacement results with realistic feeling, and can process large motion, light and shade illumination and various styles of clothing. Compared with the past methods, the present application first restores high-quality pixel-wise UV based on specific texture, and realizes high-quality texture replacement.

[0123] On the other hand, corresponding to the aforementioned monocular video clothing texture replacement method based on grid texture, the embodiment of the present application also proposes a monocular video clothing texture replacement system based on grid texture, which comprises one or more processors for implementing the monocular video clothing texture replacement method based on grid texture in the above embodiment.

[0124] ​The embodiment of the monocular video clothing texture replacement system based on grid texture can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The apparatus embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for running by the processor of the device with data processing capability. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, the device with data processing capability in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, which will not be described here.

[0125] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, which will not be described here.

[0126] For the apparatus embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The apparatus embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0127] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the monocular video clothing texture replacement method based on grid texture in the above embodiment.

[0128] The computer readable storage medium can be an internal storage unit of any device with data processing capability, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any device with data processing capability and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the device with data processing capability, and can also be used to temporarily store data that has been output or will be output.

[0129] Those skilled in the art can understand that the above description is only the preferred examples of the application and is not used to limit the application, and although the application is described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions recorded in the foregoing examples or make equivalent replacements for part of the technical features. Any modification, equivalent replacement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for replacing clothing textures in monocular video based on mesh textures, characterized in that, Includes the following steps: Step 1: Simulate a mesh textured cloth using a physics simulator and render it using a renderer to obtain the result, which will be used as training data for synthesis. Step 2: Construct and train a grid network, color network, shadow network, and UV gradient network based on the U-Net architecture. The inputs to the grid network, color network, and shadow network are all H×W×3 original images. The output of the grid network is an H×W×3 image, where each pixel contains three channels, representing the probability that the pixel is the center point of the grid, the probability that the pixel is a corner point of the grid, and the probability that the pixel is another type. The output of the color network is an H×W×8 image, where each pixel contains eight channels, representing the probability that the pixel is one of the eight colors of the grid texture. The eight colors include red, blue, yellow, green, cyan, purple, white, and black, with black being the boundary color of the grid and the others being the color of the grid itself. The output of the shadow network is an H×W×1 image, i.e., the shadow layer. The inputs to the UV gradient network are the H×W×3 original image and H×W×4 UV gradient directions, and the output is the H×W×4 UV gradient. Step 3: Using fabric with the same mesh texture as in Step 1, cut out the garment and have it worn by a model, film the model's movements, and generate a video; Step 4: Input each frame of the video from Step 3 into the trained grid network, color network, shadow network, UV gradient network, and optical flow prediction network respectively. Extract the corner points, center points, and color category information of the clothing part of the image frame by frame, as well as the shadow layer, UV gradient, and forward and backward optical flow between two adjacent frames of the video. Based on the corner points, center points, and color category information of the clothing part of each frame, use the corresponding point detection algorithm to identify the corresponding UV points. Step 5: Construct and train a hybrid parameter multilayer perceptron (HLP). The HLP includes a dimension-enhancing layer based on random Fourier features and four hidden layers of size 256. The HLP defines a set of MLP parameters for each keyframe of the video, and the MLP parameters change continuously with time t. The input of the HLP is the sequence number of each frame and the pixel coordinates of that frame, and the output is the UV coordinates. When training the HLP, UV correspondence point constraints and temporal constraints are set. Step 6: Input each frame of the video obtained in Step 3 into the trained hybrid parameter multilayer perceptron to obtain the UV coordinates of each pixel in each frame and the UV gradient direction; input the UV gradient direction and each frame of the video into the trained UV gradient network to extract the UV gradient of the clothing part frame by frame; based on the UV gradient, train the hybrid parameter multilayer perceptron again and apply additional UV gradient constraints to obtain the hybrid parameter multilayer perceptron after secondary training. Step 7: Obtain the UV coordinates of each pixel in each frame using the hybrid parameter multilayer perceptron after secondary training. Combine this with the shadow layer of the clothing part extracted frame by frame obtained in Step 3 to obtain the clothing texture replacement result with realistic shadow effect.

2. The method for replacing clothing texture in monocular video based on mesh texture according to claim 1, characterized in that, Step one includes the following sub-steps: (1.1) Construct a piece of cloth in the simulator, place it on an irregular geometry, drive the irregular geometry to move, drive the cloth to move, and obtain the cloth simulation sequence; (1.2) Attach a mesh texture to the cloth simulation sequence in the renderer and render the original image, shadow layer, mesh corners and center points, mesh color category, and UV coordinates.

3. The method for replacing clothing texture in monocular video based on mesh texture according to claim 2, characterized in that, Apply a Sobel convolution kernel to the obtained UV coordinates to obtain the UV gradient; normalize the magnitude of the UV gradient to obtain the UV gradient direction.

4. The method for replacing clothing texture in monocular video based on mesh texture according to claim 1, characterized in that, In step four, based on the corner points, center points, and color category information of the clothing portion of each frame image, a corresponding point detection algorithm is used to identify UV corresponding points, specifically including the following sub-steps: (1) Connect the corner points with the center point to obtain a heterogeneous graph, and connect the center points with each other to obtain a homogeneous graph; (2) Extract the 3x3 grid color from the isomorphic map and obtain the corresponding UV points; (3) A multi-mesh texture voting algorithm is used to determine the mesh texture of each UV point; The UV correspondence point growth algorithm is adopted to search for more UV correspondence points starting from the boundary of the isomorphic graph.

5. The method for replacing clothing texture in monocular video based on mesh texture according to claim 1, characterized in that, In step five, for a video with a total of T frames, every s frames is a keyframe, and a total of [number] keyframes are defined. MLP parameters; MLP parameters θ i This represents the i-th MLP parameter, defined over time i*s; For frame t, the parameters of the hybrid parameter multilayer perceptron are defined as a parameter set. Linear combination of parameters; hybrid parameter multilayer perceptron It has the following forms: Where α is a linear combination parameter, calculated using cubic spline differences; x is the pixel coordinate; and f is the multilayer perceptron network.

6. The method for replacing clothing texture in monocular video based on mesh texture according to claim 5, characterized in that, When training the hybrid parameter multilayer perceptron, the UV corresponding point constraint L is set. data for: in, Let be the pixel coordinates and UV of the i-th corresponding point in frame t.

7. The method for replacing clothing texture in monocular video based on mesh texture according to claim 6, characterized in that, The temporal constraints include repair constraints and continuity constraints; the UV coordinates covered by the isomorphic graph are taken as accurate UV coordinates, and the UV coordinates not covered are taken as inaccurate UV coordinates. The repair constraint specifically involves extracting the accurate UV coordinates in adjacent frames and repairing the inaccurate UV coordinates in the current frame.

8. A monocular video clothing texture replacement system based on mesh texture, characterized in that, The system includes one or more processors for implementing the monocular video clothing texture replacement method based on mesh texture as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the monocular video clothing texture replacement method based on mesh texture as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the monocular video clothing texture replacement method based on mesh texture as described in any one of claims 1 to 7.

Images