Training method of multi-task model and virtual trying-on method of clothes

Abstract

The present application provides a multi-task model training method and a virtual clothing try-on method. In the virtual clothing try-on method, by obtaining a user's body image and an image of clothing to be tried on, the user's body image and the image of clothing to be tried on are input into the multi-task model to obtain a try-on result of at least one body posture of the user. In this solution, the multi-posture body generation model and the try-on model in the prior art are replaced by a multi-task model, which reduces the occupied computing resources, improves the computing speed and the efficiency of generating try-on results, and displays the comprehensive effect of clothing from the perspective of multiple postures, providing users with more reference value. In actual scenarios, it is more in line with the user's reference habits and can improve product viscosity.

Description

Technical Field

[0001] The present application relates to the field of computer graphics and image technology, and in particular to a multi-task model training method and a clothing virtual try-on method. Background Art

[0002] With the popularity of e-commerce platforms and the development of graphics and image processing technology, the demand for online clothing shopping has increased significantly, and virtual try-on technology based on graphics and image processing has also shown huge market potential and business prospects. Virtual try-on can simulate the effect of dressing for users, thereby helping users to choose suitable clothes, promote transactions, and reduce the return rate of online clothing shopping. Therefore, how to achieve virtual try-on is the key.

[0003] At present, the realization of virtual try-on mainly involves inputting the user's image into a multi-pose human body generation model, obtaining multiple multi-pose human body images output by the human body generation model, and inputting the multiple multi-pose human body images and images of clothes to be tried on into the try-on model, obtaining the user's try-on images in multiple poses output by the try-on model, thereby achieving the purpose.

[0004] However, in the above methods, the multi-pose human body generation model and the fitting model are mostly complex in structure. Both models require more computing resources and have a slow computing speed, which results in low efficiency in generating user fitting images in multiple poses. Summary of the invention

[0005] The present application provides a multi-task model training method and a virtual clothing try-on method to solve the problem that the two models require more computing resources and have a slow computing speed, which leads to low efficiency in generating user try-on images in multiple postures.

[0006] In a first aspect, the present application provides a multi-task model training method and a clothes virtual try-on method, including:

[0007] Generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image;

[0008] Model training is performed according to the sample image set to obtain a multi-task model, wherein the multi-task model includes a first sub-task based on a spatial attention mechanism and a channel attention mechanism, and a second sub-task based on a residual network, and the weight of the loss function of each sub-task is determined according to the homoscedastic uncertainty between the sub-tasks. The multi-task model is used to generate a fitting result of at least one body posture of the user based on the user's body image and the image of the clothes to be tried on.

[0009] In a possible design of the first aspect, before performing model training according to the sample image set to obtain the multi-task model, the method further includes:

[0010] Pre-train the ImageNet dataset to obtain model parameters;

[0011] According to the model parameters, the initial parameters in the model are updated before model training.

[0012] In another possible design of the first aspect, the sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes the at least one user body sample image.

[0013] In a second aspect, an embodiment of the present application provides a virtual clothing try-on method, comprising:

[0014] Obtaining a user's body image and an image of clothes to be tried on;

[0015] The user's body image and the image of the clothes to be tried on are input into a multi-task model to obtain a trying-on result of at least one body posture of the user. The multi-task model is obtained by model training based on at least one user body sample image and at least one clothing sample image corresponding to the body posture in each user body sample image. The multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network. The weight of the loss function of each subtask is determined based on the homoscedastic uncertainty between the subtasks.

[0016] In a possible design of the second aspect, the multi-task model includes a fusion module, which is used to fuse the sub-try-on results of the first subtask and the sub-try-on results of the second subtask to generate a try-on result of at least one human posture of the user.

[0017] In a third aspect, an embodiment of the present application provides a training device for a multi-task model, comprising:

[0018] A generating module, configured to generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image;

[0019] A training module is used to perform model training according to the sample image set to obtain a multi-task model, wherein the multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network, wherein the weight of the loss function of each subtask is determined according to the homoscedastic uncertainty between the subtasks, and the multi-task model is used to generate a fitting result of at least one body posture of the user according to the user's body image and the image of the clothes to be tried on.

[0020] In a possible design of the third aspect, before performing model training according to the sample image set to obtain the multi-task model, the training device of the multi-task model further includes:

[0021] The training module is also used to pre-train the ImageNet dataset to obtain model parameters;

[0022] The updating module is used to update the initial parameters in the model according to the model parameters before the model is trained.

[0023] In another possible design of the third aspect, the sample image set includes sample image subsets for each subtask, the sample image subset for the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset for the second subtask includes the at least one user body sample image.

[0024] In a fourth aspect, an embodiment of the present application provides a virtual clothing fitting device, comprising:

[0025] An acquisition module is used to acquire a user's body image and an image of clothes to be tried on;

[0026] An input module is used to input the user's body image and the image of the clothes to be tried on into a multi-task model to obtain the trying-on result of at least one body posture of the user. The multi-task model is obtained by model training based on at least one user body sample image and at least one clothing sample image corresponding to the body posture in each user body sample image. The multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network. The weight of the loss function of each subtask is determined based on the homoscedastic uncertainty between the subtasks.

[0027] In a possible design of the fourth aspect, the multi-task model includes a fusion module, which is used to fuse the sub-try-on results of the first subtask and the sub-try-on results of the second subtask to generate a try-on result of at least one human posture of the user.

[0028] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor is used to implement the methods provided by the first aspect, the second aspect, and various possible designs when executing the computer program instructions.

[0029] In a sixth aspect, an embodiment of the present application may provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the first aspect, the second aspect, and various possible designs.

[0030] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the methods provided by the first aspect, the second aspect, and various possible designs.

[0031] The training method of the multi-task model and the virtual try-on method of clothes provided in the embodiment of the present application, in the virtual try-on method of clothes, by obtaining the user's body image and the image of the clothes to be tried on, the user's body image and the image of the clothes to be tried on are input into the multi-task model, and the try-on result of at least one body posture of the user is obtained. In this scheme, the multi-task model is used to replace the multi-posture human body generation model and the try-on model in the prior art, which reduces the occupied computing resources, improves the computing speed and the efficiency of generating try-on results. In addition, the multi-task model is used to achieve that in the case of occlusion, the clothes in the try-on results can fit the target human body as much as possible and maintain the texture details, solves the artifact phenomenon after the human body produces various postures and causes the clothes to be deformed, and then displays the comprehensive effect of the clothes from the perspective of multiple postures, providing more reference value for users. In actual scenarios, it is more in line with the user's reference habits and can improve product viscosity. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 A schematic diagram of an application scenario of the virtual try-on method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a first embodiment of a method for training a multi-task model provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of a second embodiment of a multi-task model training method provided in an embodiment of the present application;

[0036] Figure 4A schematic diagram of a flow chart of a first embodiment of a virtual clothing try-on method provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a flow chart of a second embodiment of a virtual clothing try-on method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of a training device for a multi-task model provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a virtual clothing try-on device provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0041] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] Before introducing the embodiments of the present application, the application background of the embodiments of the present application is first explained:

[0044] In recent years, the online fashion industry has been influenced by the latest technological innovations such as augmented reality, virtual reality, wearable technology and connected fitting rooms, which have provided strong support for creating new product catalogs and providing shoppers with an immersive fitting environment, influencing e-stores and opening up new possibilities for easy image editing. Virtual try-on can simulate the effect of dressing for users, thereby helping users to choose suitable clothes, promote transactions and reduce the return rate of online clothing purchases. Therefore, how to achieve virtual try-on is the key.

[0045] At present, virtual try-on is mainly achieved through multi-pose human body generation model and try-on model. Specifically, the user's image is input into the multi-pose human body generation model, and multiple multi-pose human body images output by the human body generation model are obtained. The multiple multi-pose human body images and the images of clothes to be tried on are input into the try-on model, and the try-on model outputs the user's try-on images in multiple poses, thereby achieving the purpose.

[0046] Exemplarily, the multi-pose human generative model existing in the prior art can be a deep generative model that can transform an image of a person from a given pose to a new pose while maintaining the consistency of fashion items. The model uses a generator and two discriminators for image synthesis. One discriminator distinguishes the generated image from the real image (training sample), and the other discriminator verifies the appearance consistency between the target pose and the generated image.

[0047] Furthermore, the existing fitting models in the prior art may include the following:

[0048] 1), Adaptive Content Generation and Preserving Network (ACGPN). The network first pre-tests the semantic layout of the reference image that will change after wearing, and then determines whether the image content needs to generate or preserve the semantic layout based on the prediction, thereby achieving realistic try-on and rich clothing details.

[0049] 2) Cloth Flow, a generative model based on appearance flow, can synthesize wearers for image generation and virtual try-on for positioning guides. Cloth Flow effectively simulates geometric changes by estimating dense flows between source and target clothing regions, naturally changing appearance, and synthesizing new images. The network has been extensively experimented on the Deep Fashion dataset and the VITON dataset for generating character images and virtual trials, respectively. Qualitative and quantitative results verify the effectiveness of the method.

[0050] 3), Pose-Normalized and Appearance-Preserved-Generative Adversarial Networks (PNAP-GAN). This network uses clothing-agnostic representations (i.e., clothing landmarks and semantic analysis graphs) as conditions and separates shape and appearance synthesis in a coarse-to-fine framework. In addition, a semantic embedding loss is introduced to guide the domain transfer of the semantic layer (i.e., maintaining clothing attributes).

[0051] However, as can be seen from the above examples, the multi-pose human body generation model and the fitting model are mostly complex in structure. Both models require more computing resources and have a slow computing speed, which results in low efficiency in generating user fitting images in multiple poses.

[0052] In view of the above problems, the invention of this application is as follows: Since the prior art requires the user's image to be input into the multi-pose human body generation model first, and the output of the multi-pose human body generation model is used as the input of the fitting model, the efficiency of generating the user's fitting images in multiple poses is low. Based on this, the inventors found that if a multi-task model can be obtained in advance, the multi-task model can simultaneously perform the tasks corresponding to the multi-pose human body generation model and the fitting model, which can solve the problem of low efficiency in generating fitting images in the prior art, thereby reducing the occupied computing resources and improving the computing speed.

[0053] For example, the virtual try-on method for clothes provided in the embodiment of the present application can be applied to Figure 1 In a schematic diagram of an application scenario shown. Figure 1 A schematic diagram of an application scenario of the virtual try-on method provided in the embodiment of the present application is used to solve the above technical problems. Figure 1 As shown, the application scenario may include: an electronic device 11 , a terminal device 12 , and a data storage device 13 .

[0054] In an embodiment of the present application, the electronic device 11 can receive human body images and images of clothes to be tried on input by the user through the terminal device 12, and can also obtain the above-mentioned human body images and images of clothes to be tried on through the network, and store them in the data storage device 13, so that the above-mentioned human body images and clothes to be tried on can be directly used when they are subsequently input into the multi-task model.

[0055] It should be understood that the electronic device 11 can receive human body images input by the user through the terminal device 12, and obtain images of clothes to be tried on through the network. It can also receive human body images input by the user through the terminal device 12, and obtain human body images through the network. The embodiments of the present application do not impose specific restrictions on this.

[0056] Optionally, the electronic device 11 can obtain the multi-task model through the network, and can also perform model training in advance based on a sample image set to obtain the multi-task model, or can obtain the multi-task model from other databases storing multi-task models. The embodiments of the present application do not impose specific restrictions on this.

[0057] It is understandable that the execution subject of the embodiment of the present application can be a terminal device, such as a mobile phone, a computer, a tablet computer, etc., or a server, such as a background processing platform, etc. Therefore, the present embodiment is explained by collectively referring to the terminal device and the server as an electronic device, and whether the electronic device is specifically a terminal device or a server can be determined according to actual conditions.

[0058] The technical solution of the present application is described in detail below through specific embodiments.

[0059] It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 2 The flowchart of the first embodiment of the training method of the multi-task model provided in the embodiment of the present application is as follows. Figure 2 As shown, the training method of the multi-task model may include the following steps:

[0061] S21. Generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image.

[0062] In a possible implementation, the electronic device adds at least one sample image of the user's body and at least one sample image of clothing corresponding to each sample image of the user's body to the same image set, and determines the image set after the addition as the sample image set. In other words, the sample image set includes at least one sample image of the user's body and at least one sample image of clothing corresponding to the human posture in each sample image of the user's body.

[0063] Among them, since users have various body postures in normal life and work, the status of clothes is different in different postures, such as the occlusion or wrinkle of clothes in different body postures. Therefore, the mapping relationship between the user's body sample image and the clothing sample image can be obtained in advance, so as to facilitate the subsequent training of the model according to the above mapping relationship and the sample image set.

[0064] Among them, the user in at least one user body sample image can be the same user or different users, and the clothing sample image corresponding to each user body sample image can be the same piece of clothing, such as dresses of different colors or sizes, or different clothes, such as short sleeves, denim jackets, sweatshirts, etc., which can be determined according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0065] Exemplarily, the mapping relationship between the above user body sample images and clothing sample images can be represented by Table 1:

[0066] Table 1

[0067]

[0068] It can be seen from Table 1 that human sample image 1 corresponds to clothing sample image 1, and human sample image 2 corresponds to clothing sample image 2 and clothing sample image 3. It should be understood that in actual application, the above mapping relationship may also include other forms or contents, and the embodiments of the present application do not specifically limit this.

[0069] Optionally, the above-mentioned mapping relationship can be a mapping relationship sent by the user's terminal device and received in advance by the electronic device, it can be a mapping relationship obtained by the electronic device from the network, or it can be pre-constructed based on the user's body sample image and clothing sample image. The embodiments of the present application do not impose specific restrictions on this.

[0070] Optionally, the user body sample images and clothing sample images may be sample images that the electronic device pre-acquires and stores from a sample database, or may be sample images that the electronic device pre-receives and inputs from the user through a terminal device. The embodiments of the present application do not impose specific restrictions on this.

[0071] Optionally, at least one sample image of the user's body includes at least one body posture. As a preferred solution, the number of the body postures may be 3.

[0072] S22. Perform model training based on the sample image set to obtain a multi-task model.

[0073] Among them, multi-task learning is a machine learning method corresponding to single-task learning. In the field of machine learning, the standard algorithm theory is to learn one task at a time, that is, when the system output is a real number. First, the complex learning problem is decomposed into theoretically independent sub-problems, and then each sub-problem is learned separately, and finally the mathematical model of the complex problem is established by combining the learning results of the sub-problems. Multi-task learning is a joint learning, where multiple tasks are learned in parallel and the learning results influence each other.

[0074] At the same time, multi-task learning is a kind of inductive transfer learning method. Its main task uses the domain-specific information contained in the training signal of related tasks to improve the generalization effect of the main task by inductive bias.

[0075] Among them, the multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network. The weight of the loss function of each subtask is determined according to the homoscedastic uncertainty between the subtasks. The multi-task model is used to generate a fitting result of at least one body posture of the user based on the user's body image and the image of the clothes to be tried on.

[0076] The first subtask is used to implement the function of fitting the model, and the second subtask is used to implement the function of generating a multi-pose human body model. It should be understood that the first subtask and the second subtask can also implement other functions, which can be limited according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0077] The generator of the second subtask includes an image encoder, a posture encoder and a corresponding decoder. The image encoder and the posture encoder are implemented based on a residual network. The image encoder and the posture encoder can provide good visual and geometric representations, and their corresponding decoders can generate pixel-realistic images. Optionally, the images generated by the generator will guide the model training through the discriminator. Compared with the deep convolutional network, this technical solution improves the simplicity of the multi-task model, further reduces the training time of the multi-task model, and improves the model training efficiency.

[0078] The training method of the multi-task model provided in the embodiment of the present application generates a sample image set based on at least one user body sample image and at least one clothing sample image corresponding to the human body posture in each user body sample image, and performs model training based on the sample image set to obtain a multi-task model. The technical solution reduces the model training time and improves the model training efficiency by training the first subtask and the second task simultaneously. In addition, the weight of the loss function of each subtask is determined based on the homoscedastic uncertainty between the subtasks, that is, the noise parameters of another subtask are learned in the loss function of each subtask, thereby improving the sharing ability between each subtask, as well as the accuracy and generalization ability of the multi-task model.

[0079] Figure 3 The flowchart of the second embodiment of the training method of the multi-task model provided in the embodiment of the present application is as follows. Figure 3 As shown, based on any of the above embodiments, before performing model training according to the sample image set to obtain the multi-task model, the training method of the multi-task model may include the following steps:

[0080] S31. Pre-train the ImageNet dataset to obtain model parameters.

[0081] The ImageNet dataset is a large-scale image dataset established to promote the development of computer image recognition technology. It is a benchmark for evaluating the performance of image classification algorithms. The ImageNet dataset includes more than 10 million images, and each image is manually labeled. The images in the ImageNet dataset cover most of the image categories that we see in our daily lives.

[0082] The pre-training method may be an existing method, and the embodiments of the present application do not impose any specific restrictions on this.

[0083] S32. According to the model parameters, the initial parameters in the model are updated before the model is trained.

[0084] In one achievable manner, the electronic device may transfer the model parameters to the model in the form of knowledge transfer to initialize the initial parameters.

[0085] In the above embodiment, the model parameters are obtained by pre-training the ImageNet data set, and the initial parameters in the model are updated before the model training according to the model parameters. In this solution, by updating the initial parameters in the model, more abundant feature information is obtained, thereby improving the efficiency and accuracy of subsequent model training.

[0086] Optionally, in some embodiments, the sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes at least one user body sample image and at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes at least one user body sample image. The embodiment of the present application simultaneously trains the first subtask and the second subtask through the sample image subset of the first subtask and the sample image subset of the second subtask, respectively, to improve the model training efficiency.

[0087] After obtaining the above multi-task model, the multi-task model can be used to virtually try on the clothes to be tried on. The following is a detailed description of the method for virtually trying on the clothes to be tried on using the multi-task model in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0088] In specific implementation, the execution subject of the virtual clothing try-on method can also be an electronic device with processing capabilities such as a terminal or a server. It should be understood that the electronic device of the virtual clothing try-on method and the electronic device executing the multi-task model training method can be the same device or different devices.

[0089] Figure 4 This is a flow chart of the first embodiment of the virtual trying-on method for clothes provided in the present application. Figure 4 As shown, the virtual trying-on method of the clothes may include the following steps:

[0090] S41, obtaining a user's body image and an image of clothes to be tried on.

[0091] Among them, the electronic device can receive the human body image and the image of clothes to be tried on sent by the user through the terminal device, can obtain the human body image and the image of clothes to be tried on through the network, and can also obtain the user's human body image and the image of clothes to be tried on from other data storage devices.

[0092] Exemplarily, the electronic device can also receive a try-on instruction sent by the user through the terminal device, and the try-on instruction includes the user's body image and the identification of the clothes to be tried on. The electronic device obtains the image of the clothes to be tried on corresponding to the identification of the clothes to be tried on from the pre-stored images based on the identification of the clothes to be tried on, thereby obtaining the above-mentioned user's body image and the image of the clothes to be tried on.

[0093] Optionally, the electronic device may also obtain at least two user body images and images of clothing to be tried on corresponding to each user body image, wherein the at least two user body images may be body images of the same user or body images of different users. In other words, the present application does not limit the number of user body images and images of clothing to be tried on, which may be determined according to actual needs, and will not be elaborated herein.

[0094] S42: Inputting the user's body image and the image of the clothes to be tried on into the multi-task model to obtain a try-on result of at least one body posture of the user.

[0095] Among them, the multi-task model is obtained by model training based on at least one user body sample image and at least one clothing sample image corresponding to the human body posture in each user body sample image. The multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network. The weight of the loss function of each subtask is determined based on the homoscedastic uncertainty between the subtasks.

[0096] The first subtask includes a deformable deep convolutional neural network (CNN), a CNN based on geometric matching, and a human feature generator. The first subtask improves the ability of the first subtask to extract feature information through a spatial attention mechanism and a channel attention mechanism, so that the fitting results generated by the multi-task model have a better fitting effect. The first subtask deforms the image of the clothing to be tried on through a deformable CNN, distorts the appearance of the clothing to be tried on into a synthetic human body analysis image, obtains the segmentation feature map corresponding to the human body analysis image through the human feature generator, and inputs the above segmentation feature map into the CNN based on geometric matching for synthesis, thereby outputting the sub-fitting result of the first subtask, thereby solving the problem of insufficient clarity and artifacts of the fitting results caused by human body occlusion and clothing distortion.

[0097] Among them, the human feature generator can be realized by Generative Adversarial Networks (GAN). Since GAN was proposed in 2014, it has attracted more and more attention from academia and industry. With the rapid development of GAN theory and models, its application in computer vision, natural language processing, human-computer interaction and other fields has become more and more in-depth, and has been continuously expanded to other fields. Among them, GAN has achieved great success in image generation, which depends on GAN's continuous improvement of modeling capabilities under the game, and finally achieved the generation of fake images. At the same time, image-to-image conversion is defined as the problem of converting the possible representation of one scene to another, such as mapping the image structure map to the RGB image, or vice versa. This problem is related to style transfer, which uses content images and style images and outputs style images with content images and style images. Image-to-image conversion can be regarded as a generalization of style transfer, because it is not limited to converting the style of the image, but can also manipulate the attributes of the object.

[0098] Optionally, the multi-task model includes a fusion module, which is used to fuse the sub-try-on results of the first subtask and the sub-try-on results of the second subtask to generate a try-on result of at least one human posture of the user, thereby improving the accuracy and visual effect of the try-on results.

[0099] In an embodiment of the present application, by obtaining a human body image of a user and an image of clothes to be tried on, the human body image of the user and the image of clothes to be tried on are input into a multi-task model to obtain a fitting result of at least one human body posture of the user. In this solution, the multi-posture human body generation model and the fitting model in the prior art are replaced by a multi-task model, thereby reducing the occupied computing resources, improving the computing speed and the efficiency of generating fitting results. In addition, the multi-task model is used to achieve that in the case of occlusion, the clothes in the fitting results can fit the target human body as closely as possible and maintain the texture details, thereby solving the artifact phenomenon caused by the deformation of clothes after the human body produces various postures, and then showing the comprehensive effect of clothes from the perspective of multiple postures, providing users with more reference value. In actual scenarios, it is more in line with the user's reference habits and can improve product viscosity.

[0100] Figure 5 This is a flow chart of the second embodiment of the virtual trying-on method for clothes provided in the embodiment of the present application. Figure 5 As shown, the virtual trying-on method of the clothes may include the following steps:

[0101] Step 1: Obtain the user's body image and the image of the clothes to be tried on.

[0102] Step 2: Input the user's body image and the image of the clothes to be tried on into the first subtask of the multi-task model to obtain the uncertainty factor of the first subtask.

[0103] Step 3: According to the uncertainty factor of the first subtask, determine the loss function of the first subtask, generate the fitting result of the first subtask, and jump to step 6.

[0104] Step 4: Input the user's body image into the second subtask of the multi-task model to obtain the uncertainty factor of the second subtask.

[0105] Step 5: According to the uncertainty factor of the second subtask, determine the loss function of the second subtask, generate the fitting result of the second subtask, and jump to step 6.

[0106] Step 6: Fusion the sub-try-on results of the first sub-task and the sub-try-on results of the second sub-task.

[0107] Step 7: Generate a fitting result of at least one human posture of the user.

[0108] It should be understood that in this embodiment, the execution time of step 2 is the same as that of step 4, and the execution time of step 3 is the same as that of step 5, that is, step 2 and step 4 are steps executed simultaneously, and step 3 and step 5 are steps executed simultaneously.

[0109] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0110] Figure 6 A schematic diagram of the structure of a multi-task model training device provided in an embodiment of the present application. Figure 6 As shown, the training device of the multi-task model includes:

[0111] A generating module 61, configured to generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image;

[0112] The training module 62 is used to perform model training based on the sample image set to obtain a multi-task model, wherein the multi-task model includes a first sub-task based on a spatial attention mechanism and a channel attention mechanism, and a second sub-task based on a residual network. The weight of the loss function of each sub-task is determined based on the homoscedastic uncertainty between the sub-tasks. The multi-task model is used to generate a fitting result of at least one human posture of the user based on the human body image of the user and the image of the clothes to be tried on.

[0113] In a possible design of an embodiment of the present application, before performing model training according to a sample image set to obtain a multi-task model, the training device of the multi-task model further includes:

[0114] The training module 62 is also used to pre-train the ImageNet dataset to obtain model parameters;

[0115] The updating module is used to update the initial parameters in the model according to the model parameters before the model is trained.

[0116] In another possible design of an embodiment of the present application, the sample image set includes sample image subsets for each subtask, the sample image subset for the first subtask includes at least one user body sample image and at least one clothing sample image corresponding to each user body sample image, and the sample image subset for the second subtask includes at least one user body sample image.

[0117] The multi-task model training device provided in the embodiment of the present application can be used to execute the multi-task model training method in any of the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0118] Figure 7 This is a schematic diagram of the structure of a virtual clothing fitting device provided in an embodiment of the present application. Figure 7 As shown, the virtual trying-on device of the clothes includes:

[0119] An acquisition module 71 is used to acquire a user's body image and an image of clothes to be tried on;

[0120] The input module 72 is used to input the user's body image and the image of clothes to be tried on into the multi-task model to obtain the trying-on result of at least one body posture of the user. The multi-task model is obtained by model training based on at least one user body sample image and at least one clothing sample image corresponding to the body posture in each user body sample image. The multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network. The weight of the loss function of each subtask is determined based on the homoscedastic uncertainty between the subtasks.

[0121] In a possible design of an embodiment of the present application, the multi-task model includes a fusion module, which is used to fuse the sub-try-on results of the first subtask and the sub-try-on results of the second subtask to generate a try-on result of at least one human posture of the user.

[0122] The virtual trying-on device for clothes provided in the embodiments of the present application can be used to execute the virtual trying-on method for clothes in any of the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0123] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0124] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 11 may include: a processor 81, a memory 82, and computer program instructions stored in the memory 82 and executable on the processor 81. When the processor 81 executes the computer program instructions, the multi-task model training method and / or the virtual clothing try-on method provided in any of the aforementioned embodiments are implemented.

[0125] Optionally, the above-mentioned components of the electronic device 11 may be connected via a system bus.

[0126] The memory 82 may be a separate storage unit or a storage unit integrated in the processor. The number of processors may be one or more.

[0127] Optionally, the electronic device 11 may also include an interface for interacting with other devices.

[0128] It should be understood that the processor 81 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0129] The system bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0130] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0131] The electronic device provided in the embodiments of the present application can be used to execute the multi-task model training method and / or the virtual clothing try-on method provided in any of the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0132] An embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the above-mentioned multi-task model training method and / or the virtual clothing try-on method.

[0133] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0134] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0135] An embodiment of the present application also provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when at least one processor executes the computer program, it can implement the above-mentioned multi-task model training method and / or clothing virtual try-on method.

[0136] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a multi-task model, characterized in that: include: Generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image; Performing model training according to the sample image set to obtain a multi-task model, wherein the multi-task model includes a first sub-task based on a spatial attention mechanism and a channel attention mechanism, and a second sub-task based on a residual network, wherein the weight of the loss function of each sub-task is determined according to the homoscedastic uncertainty between the sub-tasks, and the multi-task model is used to generate a fitting result of at least one human posture of the user according to the human body image of the user and the image of the clothing to be tried on; Among them, the first subtask includes a deformable deep convolutional neural network, a deep convolutional neural network based on geometric matching, and a human feature generator. The first subtask improves the ability of the first subtask to extract feature information through a spatial attention mechanism and a channel attention mechanism; the first subtask deforms the image of the clothes to be tried on through a deformable deep convolutional neural network, distorts the appearance of the clothes to be tried on into a synthetic human body analysis image, obtains a segmentation feature map corresponding to the human body analysis image through the human feature generator, and inputs the above segmentation feature map into the deep convolutional neural network based on geometric matching for synthesis, thereby outputting the sub-try-on result of the first subtask; The generator of the second subtask includes an image encoder, a posture encoder and a decoder, which are used to realize the function of generating a multi-posture human body model; wherein the image encoder and the posture encoder are implemented based on a residual network; The sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes the at least one user body sample image; The multi-task model includes a fusion module, which is used to fuse the sub-try-on result of the first subtask and the sub-try-on result of the second subtask to generate a try-on result of at least one human body posture of the user.

2. The method according to claim 1, characterized in that Before performing model training according to the sample image set to obtain the multi-task model, the method further includes: Pre-train the ImageNet dataset to obtain model parameters; According to the model parameters, the initial parameters in the model are updated before model training.

3. A virtual clothing try-on method, characterized in that: include: Obtaining a user's body image and an image of clothes to be tried on; Inputting the user's body image and the image of the clothes to be tried on into a multi-task model to obtain a fitting result of at least one body posture of the user, wherein the multi-task model is obtained by model training based on a sample image set, and the multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network, and the weight of the loss function of each subtask is determined according to the homoscedastic uncertainty between the subtasks; Among them, the first subtask includes a deformable deep convolutional neural network, a deep convolutional neural network based on geometric matching, and a human feature generator. The first subtask improves the ability of the first subtask to extract feature information through a spatial attention mechanism and a channel attention mechanism; the first subtask deforms the image of the clothes to be tried on through a deformable deep convolutional neural network, distorts the appearance of the clothes to be tried on into a synthetic human body analysis image, obtains a segmentation feature map corresponding to the human body analysis image through the human feature generator, and inputs the above segmentation feature map into the deep convolutional neural network based on geometric matching for synthesis, thereby outputting the sub-try-on result of the first subtask; The generator of the second subtask includes an image encoder, a posture encoder and a decoder, which are used to realize the function of generating a multi-posture human body model; wherein the image encoder and the posture encoder are implemented based on a residual network; The sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes the at least one user body sample image; The multi-task model includes a fusion module, and the fusion module is used to fuse the sub-try-on result of the first subtask and the sub-try-on result of the second subtask to generate a try-on result of at least one human body posture of the user.

4. A training device for a multi-task model, characterized in that: include: A generating module, configured to generate a sample image set according to at least one user body sample image and at least one clothing sample image corresponding to a body posture in each user body sample image; A training module, used for performing model training according to the sample image set to obtain a multi-task model, wherein the multi-task model includes a first sub-task based on a spatial attention mechanism and a channel attention mechanism, and a second sub-task based on a residual network, wherein the weight of the loss function of each sub-task is determined according to the homoscedastic uncertainty between the sub-tasks, and the multi-task model is used for generating a fitting result of at least one human posture of the user according to the human body image of the user and the image of the clothing to be tried on; Among them, the first subtask includes a deformable deep convolutional neural network, a deep convolutional neural network based on geometric matching, and a human feature generator. The first subtask improves the ability of the first subtask to extract feature information through a spatial attention mechanism and a channel attention mechanism; the first subtask deforms the image of the clothes to be tried on through a deformable deep convolutional neural network, distorts the appearance of the clothes to be tried on into a synthetic human body analysis image, obtains a segmentation feature map corresponding to the human body analysis image through the human feature generator, and inputs the above segmentation feature map into the deep convolutional neural network based on geometric matching for synthesis, thereby outputting the sub-try-on result of the first subtask; The generator of the second subtask includes an image encoder, a posture encoder and a decoder, which are used to realize the function of generating a multi-posture human body model; wherein the image encoder and the posture encoder are implemented based on a residual network; The sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes the at least one user body sample image; The multi-task model includes a fusion module, which is used to fuse the sub-try-on result of the first subtask and the sub-try-on result of the second subtask to generate a try-on result of at least one human body posture of the user.

5. A virtual clothing fitting device, characterized in that: include: An acquisition module is used to acquire a user's body image and an image of clothes to be tried on; An input module, used for inputting the human body image of the user and the image of the clothes to be tried on into a multi-task model, and obtaining a try-on result of at least one human body posture of the user, wherein the multi-task model is obtained by model training based on a sample image set, and the multi-task model includes a first subtask based on a spatial attention mechanism and a channel attention mechanism, and a second subtask based on a residual network, and the weight of the loss function of each subtask is determined according to the homoscedastic uncertainty between the subtasks; Among them, the first subtask includes a deformable deep convolutional neural network, a deep convolutional neural network based on geometric matching, and a human feature generator. The first subtask improves the ability of the first subtask to extract feature information through a spatial attention mechanism and a channel attention mechanism; the first subtask deforms the image of the clothes to be tried on through a deformable deep convolutional neural network, distorts the appearance of the clothes to be tried on into a synthetic human body analysis image, obtains a segmentation feature map corresponding to the human body analysis image through the human feature generator, and inputs the above segmentation feature map into the deep convolutional neural network based on geometric matching for synthesis, thereby outputting the sub-try-on result of the first subtask; The generator of the second subtask includes an image encoder, a posture encoder and a decoder, which are used to realize the function of generating a multi-posture human body model; wherein the image encoder and the posture encoder are implemented based on a residual network; The sample image set includes sample image subsets of each subtask, the sample image subset of the first subtask includes the at least one user body sample image and the at least one clothing sample image corresponding to each user body sample image, and the sample image subset of the second subtask includes the at least one user body sample image; The multi-task model includes a fusion module, which is used to fuse the sub-try-on result of the first subtask and the sub-try-on result of the second subtask to generate a try-on result of at least one human body posture of the user.

6. An electronic device comprising: A processor, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor is used to implement the method according to any one of claims 1 to 3 when executing the computer program instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 3 when executed by a processor.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 3.

Images