Cloud collaborative efficient virtual fitting method based on adaptive frame selection

Abstract

The present application relates to a kind of cloud collaborative virtual fitting method based on adaptive frame selection, it is related to the virtual fitting field based on two-dimensional image.Utilize the image information of current user that mobile phone camera real-time acquisition, according to image information in mobile phone end utilizes Resnet network to complete three-dimensional human reconstruction, obtains the parameter information of SMPL human model;The difference of current three-dimensional posture is compared with before three-dimensional posture, and according to the degree of difference to select the image and SMPL parameter is uploaded to cloud end;Cloud end recovers according to the parameter received SMPL model, and finds out corresponding body part and exchanges;Cloud end utilizes two UNet networks respectively to synthesize background and foreground, and the final output is completed after combination.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual fitting based on two-dimensional images, and in particular to a cloud collaborative efficient virtual fitting method based on adaptive frame selection. BACKGROUND

[0002] Virtual fitting is a technology application that allows users to change clothes without taking off their clothes. In recent years, with the development of generative adversarial networks, many researchers have focused on using simple two-dimensional images to complete virtual fitting. The general process of virtual fitting based on two-dimensional images is as follows: (1) human body analysis: input the human body image into the human body parser to obtain information such as body parts, face, hair, clothes and posture, and then effectively guide the synthesis of the accurate area of the human body part; (2) clothing information extraction: obtain the information of the target clothing by deformation or coding; (3) human body image synthesis: according to the above condition information, use generative adversarial network to complete the generation of the final dressing image.

[0003] Although the research on virtual fitting based on two-dimensional images has made sufficient progress, most of the methods can only run on expensive high-performance servers, and lack of interaction with users. Considering the convenience and popularity of smart phones, using smart phones to complete real-time dressing can improve the interaction of the dressing process. However, the computing power of the current mobile phone cannot meet the needs of the running of the dressing algorithm, especially for deep learning algorithms such as generative adversarial network. Therefore, cloud collaboration is an effective solution: simple tasks (human body analysis) are completed on the mobile phone, and complex and time-consuming tasks (image synthesis) are uploaded to the cloud for completion. SUMMARY

[0004] Technical problems to be solved

[0005] In order to avoid the shortcomings of the prior art, the present application provides a cloud collaborative efficient virtual fitting method based on adaptive frame selection.

[0006] Technical scheme

[0007] A cloud collaborative efficient virtual fitting method based on adaptive frame selection, characterized by the following steps:

[0008] Step 1: use the camera of the mobile phone to collect the image information of the current user in real time, and use the human body grid recovery network HMR on the mobile phone to complete the three-dimensional human body reconstruction according to the image information, to obtain the body shape parameters and posture parameters of the skinning multiple linear model SMPL And record the current parameter value K=(θ,β);

[0009] Step 2: compare the difference between the current parameter value K and the global parameter value K', if the difference is greater than a given threshold, it means that the pose change is large, the frame image I and the current parameter value K need to be uploaded to the cloud for subsequent processing, and the global parameter value K' is updated to the current parameter value K;

[0010] Step 3: the cloud receives the parameters θ, β and restores the 6890 vertices of the SMPL three-dimensional model:

[0011] N v =SMPL(θ,β)

[0012] Use the neural three-dimensional grid renderer NMR to render the three-dimensional model to obtain the two-dimensional image C of the source model under the camera perspective s , C s contains the correspondence between the three-dimensional image and the two-dimensional image in space; according to the body part to which each vertex of the SMPL three-dimensional model belongs, the input image can be divided into foreground image I fs and background image I bs , and the foreground image I fs can be further divided into different parts of the body, denoted as PARTS[face, neck,.., leg]; the reference image is processed in the same way to obtain C r , its foreground image I fr and background image I br , and the corresponding body segmentation information PARTS[face, neck,.., leg];

[0013] Mask M t is obtained by masking the clothing part of the original image; the clothing part of the reference image is replaced in the MASK part to obtain the image representation I sgt to be generated after dressing change;

[0014] Step 4: use the generative adversarial network to generate the source background image I bgs and the target image I sgt respectively:

[0015]

[0016]

[0017] Where the generator G is implemented using the UNet network;

[0018] Finally, the background image and the foreground image are linearly synthesized and weighted by the target mask M t to obtain the final image output, and the result is returned to the mobile terminal

[0019]

[0020] The further technical solution of the present application is that the difference calculation formula in step 2 is as follows:

[0021]

[0022] A computer system comprising: one or more processors; computer-readable storage media storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0023] A computer-readable storage medium storing computer-executable instructions that, when executed, implement the method described above.

[0024] Advantages

[0025] The method provided by the present application comprises the following steps: collecting image information of a current user in real time by using a mobile phone camera; completing three-dimensional human body reconstruction by using a Resnet network on a mobile phone end according to the image information, and obtaining parameter information of an SMPL human body model; comparing a difference between a current three-dimensional posture and a previous three-dimensional posture, and selecting to upload the image and the SMPL parameter to a cloud end according to a difference degree; restoring the SMPL model according to the received parameter, and finding out corresponding body parts to exchange; synthesizing the background and the foreground by using two UNet networks respectively, and completing final output after combination.

[0026] The method has the following advantages:

[0027] (1) The adjacent image frames collected by the mobile phone camera change little, and only the image frames with large posture changes are selected for processing, so that the calculation amount can be reduced while the dressing effect is maintained;

[0028] (2) The two-dimensional image is difficult to effectively describe the shape characteristics of the human body, and the three-dimensional information is introduced to describe the shape characteristics of the human body, so as to reduce the deviation;

[0029] (3) In order to maintain the background of the source image, the foreground and the background are synthesized separately, so as to generate a dressing picture consistent with the source background. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain the principles of the application, and, wherein like reference numerals refer to like parts throughout the various views.

[0031] Figure 1 The overall system architecture diagram in the present application example is shown in the figure.

[0032] Figure 2is a detailed structure diagram of the model. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0034] The present application provides a cloud-based collaborative efficient virtual fitting method based on adaptive frame selection, which is based on the following principles: using cloud combination can quickly complete virtual dressing: three-dimensional pose reconstruction is completed on the mobile phone end, rendering synthesis is completed on the cloud end, and the transformation effect is returned. In order to ensure the quality and speed of virtual fitting, the following solutions are provided: (1) the adjacent image frames collected by the mobile phone camera do not change much, only the image frames with large pose changes are selected for processing, which can reduce the computational load while maintaining the dressing effect; (2) two-dimensional images are difficult to effectively describe the shape characteristics of the human body, three-dimensional information is introduced to describe the shape characteristics of the human body, thereby reducing the deviation; (3) in order to maintain the source image background, the foreground and background are separated and synthesized, thereby generating a dressing picture consistent with the source background.

[0035] comprises:

[0036] The cloud-based collaborative virtual fitting method: three-dimensional pose reconstruction is completed on the mobile phone end, rendering synthesis is completed on the cloud end, and the transformation effect is returned;

[0037] The adaptive frame selection method for virtual fitting: compare the difference between the SMPL parameters of the current frame and the global SMPL parameters, and select whether to upload to the cloud for rendering synthesis according to the difference size.

[0038] The specific steps are as follows:

[0039] Step 1: use the mobile phone camera to collect the image information of the current user in real time, use the human mesh recovery network HMR (Human Mesh Recovery) to complete three-dimensional human body reconstruction on the mobile phone end according to the image information, and obtain the body shape parameters and pose parameters of the skinned multi-person linear model SMPL (Skinned Multi-Person Linear Model) and record the current parameter value K=(θ,β).

[0040] Step 2: At the initial time of algorithm running, set the global parameter value K' as the default value 0. At each time step, compare the difference between the current parameter value K and the global parameter value K'. If the difference is greater than a given threshold, it means that the pose change is large, and the frame image I and the current parameter value K need to be uploaded to the cloud for subsequent processing, and the global parameter value K' is updated to the current parameter value K. The difference calculation formula is as follows:

[0041]

[0042] Step 3: After the cloud receives the parameters θ, β, the 6890 vertices of the SMPL three-dimensional model are recovered: v N

[0043]

[0044] The three-dimensional human model is rendered using the neural three-dimensional mesh renderer NMR (Neural 3D Mesh Renderer) to obtain the two-dimensional image C of the source model under the camera perspective s . C s contains the spatial correspondence between the three-dimensional image and the two-dimensional image. According to the human body parts of each vertex of the SMPL three-dimensional model, the input image can be divided into foreground image I fs and background image I bs , and the foreground image I fs can be further divided into different parts of the body, denoted as PARTS[face, neck,.., leg]. The same processing is performed on the reference image to obtain C r , its foreground image I fr and background image I br , and the corresponding body segmentation information PARTS[face, neck,.., leg].

[0045] The clothing part of the original image is masked to obtain the mask M t . The clothing of the reference image is deformed and replaced in the masked area of the original image to obtain the generated image representation I sgt after dressing.

[0046] Step 4: Use the generative adversarial network to generate the source background image I bg and the target image I sg respectively:

[0047]

[0048]

[0049] Where the generator G is implemented using the UNet network. ​

[0050] Finally, the background and foreground images are linearly composited, and the target mask M is applied t The weighting is performed to obtain a final image output, and the result is returned to the mobile terminal

[0051]

[0052] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application.

Claims

1. A cloud-based collaborative efficient virtual fitting method based on adaptive framing, characterized in that The steps are as follows: Step 1: Real-time acquisition of current user image information by using mobile phone camera, three-dimensional human body reconstruction is completed by using human body grid recovery network HMR on mobile phone according to image information, and body shape parameters and posture parameters of skin multi-person linear model SMPL are obtained , and the current parameter value is recorded ; Step 2: compare the current parameter value with the global parameter value If the difference is greater than a given threshold, it means that the pose change is large, and the frame image should be uploaded to the cloud for further processing, and the global parameter value should be updated to the current parameter value . ​​ The difference calculation formula is as follows: Step 3: Cloud receives parameters Post restoration of SMPL 3D model with 6890 vertices: A neural three-dimensional mesh renderer (NMR) is used to render the three-dimensional model to obtain a two-dimensional image of the source model from the perspective of the camera , The three-dimensional image and the two-dimensional image contain a spatial correspondence relationship; according to the human body part to which each vertex of the SMPL three-dimensional model belongs, the input image can be divided into a foreground image and a background image , and the foreground image can be further divided into different parts of the body, denoted as PARTS[face, neck,.., leg]; the same processing is performed on the reference image to obtain , and its foreground image and background image , and the corresponding body segmentation information PARTS[face, neck,.., leg]; masking the clothing portion of the original image to obtain a mask ; replacing the clothing part of the reference image to the MASK part, resulting in a corresponding generated image representation after the clothes change ; Step 4: Perform generative processing on the source background image and the target image respectively using a generative adversarial network: wherein the generator implemented using a UNet network Finally, the background and foreground images are linearly composited, and the target mask is applied The weighting is performed to obtain a final image output, and the result is returned to the mobile terminal 。 2. A computer system, characterized by Comprise: One or more processors, computer readable storage media, for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors make the method of claim 1.

3. A computer-readable storage medium, characterized in that Computer executable instructions are stored, which instructions when executed are used to implement the method of claim 1.

Images