Training method of virtual reloading model and virtual reloading method and device

By extracting the character's clothing and posture information and adjusting the virtual dress-up model parameters, the problem of clothing migration error is solved, the naturalness and realism of dress-up is improved, and a personalized virtual dress-up experience is provided.

CN119091092APending Publication Date: 2024-12-06BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411073054.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing virtual dress-up method has errors when migrating the costume from one character to another with different body shapes or postures, resulting in mismatch between the costume area and the character's posture, and the generated dress-up image does not match the expectations. Not effective.

Method used

By obtaining the original image and clothing images of the characters, extracting text descriptions of clothing and non-clothing elements, as well as character posture information, inputting them into the virtual dress-up model, and adjusting the difference between the dress-up image output by the model and the original image. Model parameters to narrow the gap and obtain the target virtual dress-up model. The model can adjust the presentation method of clothing according to different text prompts and character postures, improving the naturalness and fit of dressing.

Benefits of technology

It realizes the naturalness and fit in different postures, enhances the flexibility and realism of dressing, and provides users with a richer and more personalized virtual dressing experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091092A_ABST
    Figure CN119091092A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a training method of a virtual clothes changing model, and a virtual clothes changing method and device, and the method comprises the steps: precisely extracting a clothes image of clothes worn by a person in an original image and a text describing non-clothes elements, and enabling the clothes to be attached to the body of the person, different figure postures can influence the visual effect of the clothes in the image, so that the figure postures in the original image are accurately captured, the information is used as input, the model parameters are further adjusted by comparing the difference between the clothes changing image output by the model and the original image, and the clothes changing efficiency is improved. According to the method and the device, the target virtual dressing changing model can adjust the dressing presenting mode according to different text prompts and different figure postures by adjusting the target virtual dressing changing model to reduce the gap between the dressing changing image and the original image, so that the naturalness and fitness of the dressing changing image under different postures are ensured, the flexibility and reality sense of dressing changing are greatly enhanced, and the dressing changing experience is improved. And richer and more personalized virtual clothes changing experience is provided for the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a training method for a virtual dressing model, a virtual dressing method and a device. Background Art

[0002] With the development of artificial intelligence technology, virtual dressing technology has emerged. Through virtual dressing technology, users only need to upload photos of people to match clothes to achieve online virtual dressing. However, the existing virtual dressing method is just a simple clothing replacement, that is, the clothing area in a specified image is segmented and transferred to the person in another image. In this process, the clothing area may be simply scaled or rotated. However, the body shapes or postures of different people are complex and diverse. There will be large errors when the same clothing is transferred to people of different body shapes or postures, resulting in mismatch between clothing areas and character postures, rough processing of the connection parts with the body parts of the characters, etc., resulting in the generated dressing image not meeting expectations and poor dressing effect. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a training method for a virtual dressing model, a virtual dressing method and a device, so as to improve the naturalness, flexibility and realism of virtual dressing, and provide users with a richer and more personalized virtual dressing experience. The specific technical solution is as follows:

[0004] In a first aspect of the present invention, a training method for a virtual dressing model is provided, the method comprising:

[0005] Acquire a first original image of a first person and acquire a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing;

[0006] Acquire a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image;

[0007] Inputting the first clothing image, the first text, and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model;

[0008] According to the difference between the first dress-changing image and the first original image, the model parameters of the original virtual dress-changing model are adjusted to obtain a target virtual dress-changing model.

[0009] In a possible embodiment, acquiring a first original image of a first person and acquiring a first clothing image of a first clothing includes:

[0010] Searching for a first original image of a first person and a first clothing image of the first clothing from a first preset data set;

[0011] or,

[0012] Searching for a first original image of a first person from a second preset data set;

[0013] The first original image is segmented using a semantic segmentation algorithm to obtain a first clothing image of the first clothing.

[0014] In a possible embodiment, the acquiring of first text for describing other elements in the first original image except the first clothing includes:

[0015] Determine a second text for describing all elements in the first original image by using a text description algorithm;

[0016] displaying the second text;

[0017] In response to a deletion instruction for the third text in the second text for describing the first clothing, the text indicated by the deletion instruction is deleted, and the remaining text is used as the first text.

[0018] In a possible embodiment, obtaining the first posture of the first person in the first original image includes:

[0019] Using a posture recognition algorithm to identify the first person in the first original image to obtain key points of the human body as a first posture;

[0020] or,

[0021] A first person in the first original image is identified by using a depth estimation algorithm to obtain a human body depth image as a first posture.

[0022] In a possible embodiment, the virtual dressing model includes an image feature extraction module, a text feature extraction module, a first diffusion module and a second diffusion module, and the diffusion module includes a forward diffusion submodule and a reverse diffusion submodule;

[0023] The step of inputting the first clothing image, the first text, and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model includes:

[0024] Inputting the first clothing image into the image feature extraction module to obtain a first image feature;

[0025] Inputting the first text into the text feature extraction module to obtain a first text feature;

[0026] Inputting the first posture into the reverse diffusion submodule of the first diffusion module to obtain a target posture;

[0027] The first image feature and the first text feature are input into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, and the target posture is input into the reverse diffusion submodule of the second diffusion module to obtain the predicted dressing image output by the second diffusion module as the first dressing image output by the original virtual dressing model.

[0028] In a possible embodiment, the second diffusion module further includes a linear projection submodule;

[0029] The step of inputting the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module includes:

[0030] Inputting the first image feature and the first text feature into the linear projection submodule to obtain a first linear feature;

[0031] inputting the first linear feature into a forward diffusion submodule and a reverse diffusion submodule of the second diffusion module;

[0032] The adjusting the model parameters of the original virtual dressing model to obtain the target virtual dressing model includes:

[0033] The module parameters of the image feature extraction module, or the module parameters of the image feature extraction module and the linear projection submodule are adjusted to obtain a target virtual dressing model.

[0034] In a possible embodiment, the method further includes:

[0035] Inputting target dressing data into an original virtual dressing model to obtain a target dressing image output by the original virtual dressing model, wherein the target dressing data includes the first clothing image, the first text, and the first posture, or the first clothing image and the first text, or the first clothing image and the first posture;

[0036] The step of adjusting the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image to obtain the target virtual dressing model includes:

[0037] According to the difference between the first dress-changing image and the first original image, and the difference between the first dress-changing image and the target dress-changing image, the model parameters of the original virtual dress-changing model are adjusted to obtain the target virtual dress-changing model.

[0038] In a second aspect of the present invention, a virtual dressing method is provided, the method comprising:

[0039] Acquiring data for clothing to be changed, wherein the data for clothing to be changed includes a second clothing image, a second text, and a second posture of the second clothing;

[0040] The data to be changed into is input into a target virtual dressing model, and a second dressing image outputted by the target virtual dressing model is obtained, wherein the target virtual dressing model is pre-trained using any of the training method steps for a virtual dressing model described in the first aspect, wherein in the second dressing image, a second character wears the second clothing, a posture of the second character in the second dressing image is the second posture, and the second dressing image conforms to the description of the second text.

[0041] In a third aspect of the present invention, a training device for a virtual dress-up model is provided, the device comprising:

[0042] An acquisition module, configured to acquire a first original image of a first person and a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing;

[0043] a determination module, configured to obtain a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image;

[0044] An input module, used for inputting the first clothing image, the first text and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model;

[0045] The adjustment module is used to adjust the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image to obtain a target virtual dressing model.

[0046] In a possible embodiment, the acquisition module includes:

[0047] Obtaining a first submodule, for searching a first original image of a first person and a first clothing image of the first clothing from a first preset data set;

[0048] or,

[0049] Obtain a second submodule, which is used to search for a first original image of a first person from a second preset data set;

[0050] A third submodule is obtained, which is used to segment the first original image using a semantic segmentation algorithm to obtain a first clothing image of the first clothing.

[0051] In a possible embodiment, the determining module includes:

[0052] Determine a first submodule, which is used to determine a second text for describing all elements in the first original image by using a text description algorithm;

[0053] Determine a second submodule for displaying the second text;

[0054] A third submodule is determined, which is used to respond to a deletion instruction for a third text in the second text for describing the first clothing, delete the text indicated by the deletion instruction, and use the remaining text as the first text.

[0055] In a possible embodiment, the determining module includes:

[0056] Determine a fourth submodule, which is used to use a posture recognition algorithm to identify the first person in the first original image to obtain human body key points as a first posture;

[0057] or,

[0058] A fifth submodule is determined, which is used to use a depth estimation algorithm to identify the first person in the first original image to obtain a human body depth image as a first posture.

[0059] In a possible embodiment, the virtual dressing model includes an image feature extraction module, a text feature extraction module, a first diffusion module and a second diffusion module, and the diffusion module includes a forward diffusion submodule and a reverse diffusion submodule;

[0060] The input module comprises:

[0061] An input first submodule is used to input the first clothing image into the image feature extraction module to obtain a first image feature;

[0062] An input second submodule is used to input the first text into the text feature extraction module to obtain a first text feature;

[0063] An input third submodule is used to input the first posture into the reverse diffusion submodule of the first diffusion module to obtain a target posture;

[0064] A fourth submodule is input, which is used to input the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, and input the target posture into the reverse diffusion submodule of the second diffusion module, so as to obtain the predicted dressing image output by the second diffusion module as the first dressing image output by the original virtual dressing model.

[0065] In a possible embodiment, the second diffusion module further includes a linear projection submodule;

[0066] The input fourth submodule comprises:

[0067] An input first subunit is used to input the first image feature and the first text feature into the linear projection submodule to obtain a first linear feature;

[0068] An input second sub-unit, used for inputting the first linear characteristic into a forward diffusion sub-module and a reverse diffusion sub-module of the second diffusion module;

[0069] The adjustment module comprises:

[0070] The first submodule is adjusted to adjust the module parameters of the image feature extraction module, or the module parameters of the image feature extraction module and the linear projection submodule, to obtain a target virtual dressing model.

[0071] In a possible embodiment, the device further includes:

[0072] a screening module, configured to input target dressing data into an original virtual dressing model to obtain a target dressing image output by the original virtual dressing model, wherein the target dressing data includes the first clothing image, the first text, and the first posture, or the first clothing image and the first text, or the first clothing image and the first posture;

[0073] The adjustment module comprises:

[0074] The second adjustment submodule is used to adjust the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image and the difference between the first dressing image and the target dressing image to obtain the target virtual dressing model.

[0075] In a fourth aspect of the present invention, a virtual dressing device is provided, the device comprising:

[0076] a module for acquiring data for clothing to be changed, used for acquiring data for clothing to be changed, wherein the data for clothing to be changed includes a second clothing image, a second text and a second posture of the second clothing;

[0077] A dressing image output module is used to input the data to be dressed up into a target virtual dressing model to obtain a second dressing image output by the target virtual dressing model, wherein the target virtual dressing model is pre-trained using any of the training method steps for a virtual dressing model described in the first aspect, wherein in the second dressing image, a second character wears the second clothing, a posture of the second character in the second dressing image is the second posture, and the second dressing image conforms to the description of the second text.

[0078] In a fifth aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0079] Memory, used to store computer programs;

[0080] The processor is used to implement any of the above-mentioned virtual dressing model training methods or virtual dressing methods when executing the program stored in the memory.

[0081] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the training method of the virtual dressing model or the virtual dressing method described above is implemented.

[0082] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the virtual dressing model training methods or virtual dressing methods described above.

[0083] The training method, virtual dressing method and device of the virtual dressing model provided by the embodiments of the present invention accurately extract the clothing image of the clothing worn by the first character in the first original image and the text describing the non-clothing elements. Since the clothing is attached to the body of the character, the different postures of the character will affect the visual effect of the clothing presented in the image. Therefore, the posture of the first character in the first original image is also accurately captured, and this information is used as input. Furthermore, the first original image is taken as the true value, and the model parameters are adjusted in a targeted manner by comparing the difference between the dressing image output by the model and the original image to narrow the gap between the dressing image and the original image, so that the target virtual dressing model finally obtained can adjust the presentation mode of the clothing according to different text prompts and different character postures, thereby ensuring the naturalness and fit of the dressing image in different postures, greatly enhancing the flexibility and realism of dressing, and providing users with a richer and more personalized virtual dressing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0085] Figure 1 Schematic diagram of the process of a virtual dress-changing model training method according to an embodiment of the present invention;

[0086] Figure 2 is a schematic diagram of a first original image and a first clothing image in an embodiment of the present invention;

[0087] Figure 3 is a schematic diagram of a first original image in an embodiment of the present invention;

[0088] Figure 4 is a schematic diagram of a first clothing image in an embodiment of the present invention;

[0089] Figure 5 is a first schematic diagram of a first posture in an embodiment of the present invention;

[0090] Figure 6 is a second schematic diagram of the first posture in an embodiment of the present invention;

[0091] Figure 7 This is a first schematic diagram of virtual dress-changing model training in an embodiment of the present invention;

[0092] Figure 8 This is a second schematic diagram of the virtual dress-changing model training in an embodiment of the present invention;

[0093] Fig. 9 This is a third schematic diagram of the virtual dress-changing model training in the embodiment of the present invention;

[0094] Fig.10 Schematic diagram of the process of the virtual dressing method in an embodiment of the present invention;

[0095] Fig.11 A schematic diagram of the process of training and applying a virtual dress-up model in an embodiment of the present invention;

[0096] Fig.12 A schematic diagram of applying a virtual dressing model in an embodiment of the present invention;

[0097] Fig.13 It is a structural schematic diagram of a virtual dress-changing model training device in an embodiment of the present invention;

[0098] Fig.14 Schematic diagram of the structure of a virtual dressing device in an embodiment of the present invention;

[0099] Fig.15 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0100] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0101] In the current field of image generation, especially when focusing on the niche scenario of virtual dressing, achieving precise control over the generated image effect has become a crucial technical challenge. Virtual dressing requires that the clothes can fit the target person seamlessly to create a natural and smooth visual effect.

[0102] Although the industry has widely adopted a variety of technical means, such as fine-tuning strategies based on mask redrawing, and image stitching technology implemented by generative models such as Diffusion models and GAN (Generative Adversarial Networks) to optimize the effect of virtual dressing, these methods still face many limitations in actual operation. Specifically, these technologies often find it difficult to perfectly solve the problem of natural fusion of clothing and background scenes in color, light and shadow, resulting in a clear boundary between clothing and background in the final generated dressing image, and a strong sense of stitching, which affects the overall visual authenticity and immersion.

[0103] In addition, although methods such as IP-Adapter (an adapter designed for pre-trained text-to-image diffusion models) and InstantID (an image generation and authentication technology) have achieved remarkable results in the field of face embedding and can achieve the natural integration of faces with different scenes, the application effect of these methods is still insufficient in clothing processing. As the core element in virtual dressing, the material, texture, color and dynamic matching degree with the character's posture of clothing have a profound impact on the final effect. Therefore, how to further improve the naturalness, flexibility and realism of dressing has become a key issue that needs to be solved in the current field of virtual dressing.

[0104] Based on this, the embodiment of the present invention provides a training method for a virtual dress-up model, see Figure 1 , Figure 1 The flowchart of the virtual dressing model training method in an embodiment of the present invention includes:

[0105] S101, obtaining a first original image of a first person and obtaining a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing.

[0106] S102, obtaining a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image.

[0107] S103: Input the first clothing image, the first text, and the first posture into the original virtual dressing model to obtain a first dressing image output by the original virtual dressing model.

[0108] S104: adjusting model parameters of the original virtual dressing model according to the difference between the first dressing-up image and the first original image to obtain a target virtual dressing-up model.

[0109] By applying the above embodiment, the clothing image of the clothing worn by the first character in the first original image and the text describing the non-clothing elements are accurately extracted. Since the clothing is attached to the body of the character, the different postures of the character will affect the visual effect of the clothing presented in the image. Therefore, the posture of the first character in the first original image is also accurately captured, and this information is used as input. Furthermore, with the first original image as the true value, by comparing the difference between the dressing image output by the model and the original image, the model parameters are adjusted in a targeted manner to narrow the gap between the dressing image and the original image, so that the target virtual dressing model finally obtained can adjust the presentation of clothing according to different text prompts and different character postures, thereby ensuring the naturalness and fit of the dressing image in different postures, greatly enhancing the flexibility and realism of dressing, and providing users with a richer and more personalized virtual dressing experience.

[0110] The above S101-S104 will be described exemplarily below:

[0111] In S101, it is necessary to obtain in advance a first original image of a first person wearing a first clothing item and a first clothing image of the first clothing item. In a possible embodiment, a user can manually select, crop, and edit the clothing portion in the image using professional image editing software or online tools. However, this method relies on the user's input and editing skills, increases the time cost of training a virtual dressing model, and cannot ensure the richness and diversity of image resources.

[0112] Based on this, in order to shorten the data preparation time, reduce the time cost of virtual dressing, and ensure the richness and diversity of image resources, in another possible embodiment, see Figure 2 , Figure 2 : is a schematic diagram of a first original image and a first clothing image in an embodiment of the present invention. The first original image of the first person and the first clothing image of the first clothing can be obtained by the following two methods:

[0113] Method 1: searching a first original image of a first person and a first clothing image of a first clothing from a first preset data set.

[0114] The first preset dataset is a public dataset mainly for the e-commerce field. The dataset contains two core parts: one is a large number of clothing images, which usually have a pure background and only show the clothes themselves; the other is the first original image of the first person (such as a model) wearing these clothes, that is, a full-body photo of the model wearing specific clothes.

[0115] In order to improve the data quality, the original first preset data set needs to be cleaned to obtain the first preset data set. This process is based on multiple key criteria, including but not limited to the clarity of the image (to ensure that the image details are clear and discernible), whether it contains faces (to confirm that there are people in the image and they are recognizable), and whether the overall brightness is within the normal range (to ensure that the image has a good visual effect and is not overexposed or underexposed). Through this series of screening and cleaning, more than 10,000 high-quality paired images were selected from the original first preset data set to form the first preset data set.

[0116] Each pair of paired images consists of two parts:

[0117] Part A: A clothing image with a pure background, focusing on showing the details and style of the clothes without other distracting elements.

[0118] Part B: The first original image of a model wearing the clothes in Part A. This image not only shows the effect of the clothes in real wearing state, but also contains information such as the model's body shape and posture, providing rich reference for subsequent virtual dressing operations.

[0119] Figure 2 As an example, the structure of this pair of images is intuitively shown, that is, 201 and 202 on the left are clothing images with pure backgrounds, and 203 and 204 on the right are the first original images of models wearing the clothing.

[0120] When a pure service image cannot be collected, the following method 2 may be used to obtain a first original image of the first person and a first clothing image of the first clothing.

[0121] Method 2: searching for a first original image of the first person from a second preset data set, and using a semantic segmentation algorithm to segment the first original image to obtain a first clothing image of the first clothing.

[0122] The second preset data set is a comprehensive image library, which includes multiple sets of character stills captured from different film and television works. For example, multiple sets of ancient costume stills of different characters can be captured in ancient costume dramas, or multiple sets of modern costume stills of different characters can be captured in modern dramas, etc. These captured stills constitute the second preset data set.

[0123] Find the first original image of the first person from the second preset data set, see Figure 3 , Figure 3 is a schematic diagram of a first original image in an embodiment of the present invention. The first original image here may be the aforementioned Figure 2 Any one of the first original images in Figure 3 Only the above Figure 2 Take the first original image 204 as an example.

[0124] The first original image is segmented using a semantic segmentation algorithm to obtain a first clothing image of the first clothing, wherein the semantic segmentation algorithm can be a SAM (Segment Anything Model) algorithm, an FCN (Fully Convolutional Network) algorithm, or other algorithms that can achieve semantic segmentation, which are not specifically limited here.

[0125] In order to improve the accuracy of obtaining the first clothing image by semantic segmentation, a face detection algorithm may be used to obtain the first clothing image.

[0126] See also Figure 4 , Figure 4 is a schematic diagram of a first clothing image in an embodiment of the present invention, Figure 4 The first clothing image in the image is the one that is calibrated using the SAM algorithm. Figure 3 The first clothing image is obtained by segmenting the first original image in .

[0127] Applying the above embodiment, adopting the above method 1, by directly searching from the preset data set, not only the data preparation time is greatly shortened, but also the richness and diversity of image resources are ensured, providing a solid foundation for the subsequent virtual dressing model training. On the other hand, when there is a lack of specific clothing images in the preset data set, the above method 2 can be adopted, and the original image of the first person can be finely segmented using an advanced semantic segmentation algorithm, which can automatically and accurately extract clothing images, not only overcoming the limitations of data acquisition, but also demonstrating a high degree of technical flexibility and intelligence. This method of combining preset data set search with semantic segmentation algorithm extraction not only ensures the convenience of data acquisition, reduces the time cost of virtual dressing model training, but also ensures the accuracy of clothing image extraction. After the virtual dressing model is trained, it can bring users a more personalized and high-quality virtual dressing experience.

[0128] In S102, it is necessary to obtain a first text for describing other elements in the first original image except the first clothing.

[0129] In one possible embodiment, an image recognition model can be used to perform in-depth analysis of the image to identify key elements such as clothing, characters, and backgrounds, and then a text description algorithm can be used to specifically describe the identified non-clothing elements (such as characters, backgrounds, etc.) to generate a first text. This method combines the advantages of automation and intelligence, but may require a large amount of training data to train the image recognition model to improve accuracy, increasing the cost of training the virtual dressing model.

[0130] Based on this, in order to reduce the cost of training the virtual dressing model and bring users a more personalized and high-quality virtual dressing experience, in another possible embodiment, a text description algorithm can be used to determine the second text used to describe all elements in the first original image, and then the second text can be displayed to the user. The user can directly edit the content in the second text according to his or her own needs, that is, delete the third text part that describes the first clothing. The system responds to the deletion instruction, automatically adjusts the text content, and uses the remaining text describing other elements as the first text.

[0131] The text description algorithm may be a BLIP2 (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models) algorithm or other image text description algorithms, which are not specifically limited here.

[0132] Exemplarily, the BLIP2 algorithm is used to Figure 2The first original image 204 is described in text form, and the obtained second text is "1 girl, solo, long hair, realistic, sweater, brown hair, closed eyes, skirt, arms at sides, turtleneck, lips". In response to a deletion instruction input by the user for the third text used to describe the first clothing, the third text "sweater, skirt, turtleneck" describing the first clothing in the second text is deleted, and the finally obtained first text is "1 girl, solo, long hair, realistic, brown hair, closed eyes, arms at sides, lips".

[0133] By applying the above embodiment, when the user needs to remove the description text about a certain part and only keep the required part of the description text, only a simple deletion command is required to achieve accurate modification, without the need for the user to manually enter the required part of the description text, which greatly improves the convenience and efficiency of editing. This interactive text editing method makes the text description of the image more flexible and diverse, meeting the changing needs in different scenarios. It not only simplifies the process of text description of image content and reduces the cost of virtual dress-up model training, but also gives users more customization and control power. After the virtual dress-up model training is completed, it can bring users a more personalized and high-quality virtual dress-up experience.

[0134] After the first text is obtained, it is also necessary to obtain the first posture of the first person in the first original image.

[0135] In a possible embodiment, a series of human posture templates may be predefined, and these templates may be matched with the human body of the first person in the first original image. By calculating similarity or distance measurement, the most matching template is found, thereby obtaining the human body posture, which is used as the first posture. However, this method has high requirements on the accuracy and diversity of the templates, so the cost of training the virtual dressing model is high, and it may take multiple attempts to find a suitable template, which makes the efficiency of virtual dressing model training low.

[0136] Based on this, in order to reduce the cost of virtual dress-changing model training and improve the efficiency of virtual dress-changing model training, in another possible embodiment, a posture recognition algorithm can be used to identify the first person in the first original image to obtain human body key points, which are used as the first posture. Alternatively, a depth estimation algorithm can be used to identify the first person in the first original image to obtain a human body depth image, which is used as the first posture.

[0137] When obtaining the first posture using a posture recognition algorithm, a posture recognition algorithm, such as the openpose (human posture estimation model) algorithm of ControlNet (a plug-in that helps users control the output of images), or other algorithms for recognizing human postures, can be used to recognize the first person in the first original image to extract the key point information of the person, such as the coordinate positions of the head, shoulders, elbows, wrists, hips, knees, ankles, etc. These key point information can be directly represented in the form of an image (such as a key point map displayed by a heat map or connecting lines), or converted into a coordinate list form, which is not specifically limited here. These key points together constitute the first posture of the person.

[0138] For example, taking the first posture as an example represented by an image, see Figure 5 , Figure 5 is a first schematic diagram of the first posture in an embodiment of the present invention, Figure 5 To use the openpose algorithm to Figure 2 The human body key point image obtained after identifying the first original image 204 clearly depicts the posture of the person, and this image is a representation of the first posture.

[0139] When the first posture is obtained using a depth estimation algorithm, a depth estimation algorithm, such as the depth (depth map model) algorithm of ControlNet, or other algorithms for obtaining a human depth map, can be used to identify the first person in the first original image. The depth estimation algorithm estimates the distance from each pixel in the image to the camera by analyzing the color, texture, and geometric information in the image, thereby generating a human depth image. This depth image reflects the shape of the person in three-dimensional space and can be regarded as another form of the first posture.

[0140] For example, see Figure 6 , Figure 6 is a second schematic diagram of the first posture in an embodiment of the present invention, Figure 6 To use the depth algorithm for the above Figure 2 The human body depth image obtained after identifying the first original image 204, this human body depth image represents the distance between each part of the person and the camera through the change of color depth, thereby constituting another form of the first posture.

[0141] By applying the above embodiments, it is possible to flexibly choose to use a posture recognition algorithm or a depth estimation algorithm to obtain the first posture to adapt to different application scenarios and needs. Both methods provide accurate information about the character's posture, which not only helps to reduce resource consumption during the training of the virtual dressing model, thereby reducing training costs, but also significantly speeds up the training process and improves training efficiency. Ultimately, these optimizations will be directly reflected in the performance of the trained virtual dressing model, enabling it to respond more accurately to various dressing needs.

[0142] In S103, after obtaining the first clothing image, the first text and the first posture, they need to be input into the original virtual dressing model, so as to obtain the first dressing image output by the original virtual dressing model.

[0143] With the first clothing image as mentioned above Figure 2 The first clothing image 202 in the figure, the first posture 703 is the aforementioned Figure 5 Take the first posture shown as an example, see Figure 7 , Figure 7 This is a first schematic diagram of virtual dress-changing model training in an embodiment of the present invention. Figure 7 In the example, the first clothing image 701, the first text 702 and the first posture 703 are input to the original virtual dress-up model 700, and the original virtual dress-up model 700 outputs the first dress-up image 704 after the dress-up.

[0144] Among them, the first clothing image 701 is the aforementioned Figure 2 The first clothing image 202 in the figure, the first text 702 is the BLIP2 algorithm for the aforementioned Figure 2 The first text obtained by performing text description on the first original image 204 is shown in Chinese only. The first posture 703 is the aforementioned Figure 5 The first pose is shown.

[0145] In a possible embodiment, the virtual dressing model includes an image feature extraction module, a text feature extraction module, a first diffusion module and a second diffusion module, wherein the diffusion module includes a forward diffusion submodule and a reverse diffusion submodule.

[0146] On this basis, the first clothing image, the first text and the first posture are input into the original virtual dressing model in S103 to obtain the first dressing image output by the original virtual dressing model, which may specifically include:

[0147] The first clothing image is input into the image feature extraction module to obtain the first image feature.

[0148] The first text is input into a text feature extraction module to obtain a first text feature.

[0149] The first posture is input into the reverse diffusion submodule of the first diffusion module to obtain the target posture.

[0150] The first image feature and the first text feature are input into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, and the target posture is input into the reverse diffusion submodule of the second diffusion module to obtain the predicted dressing image output by the second diffusion module as the first dressing image output by the original virtual dressing model.

[0151] When the first clothing image and the first posture in image form are input into the original virtual dressing model 700 , they need to be cropped according to the versions of the first diffusion module and the second diffusion module in the original virtual dressing model 700 .

[0152] For example, when the first diffusion module and the second diffusion module are Stable Diffusion models, the input image (especially the main content of the image) needs to be cropped to a size of 512x512 pixels. This is because the input size of the Stable Diffusion 1.x model is usually designed to receive an image of 512x512 pixels to ensure that the model can effectively process and generate images of the corresponding size.

[0153] When the first diffusion module and the second diffusion module are stable diffusion XL, the main content of the image needs to be cropped to a size of 1024x1024 pixels. This is because the stable diffusion XL version supports larger input sizes to fully utilize its enhanced generation capabilities.

[0154] In addition to Stable Diffusion, there are many other image generation models based on the diffusion principle. These models may use different architectures, such as DiT (Diffusion Transformer, a diffusion model combined with an attention mechanism) or Dima (an image generation model). When using these models for training, you need to follow their specific requirements for the input image size and crop and process them accordingly.

[0155] The aforementioned image feature extraction module may include a CLIP model (Contrastive Language-Image Pre-training, a multimodal model based on contrastive learning), such as an Image Encoder, and a separate Image Projection Model, such as a Linear layer and an LN (Layer Normalization). The image encoder in the image feature extraction module extracts features from the first clothing image to obtain image features, and the linear layer and the normalization layer extract the extracted image features. In actual operation, an architecture combined with a Transformer (a neural network model based on a self-attention mechanism) is used to design the linear layer and the normalization layer to extract clothing information from the image features, and the amount of extracted image features (i.e., the amount of information contained in the original clothing) can be set in advance. For example, if all image features are used, the original clothing information is restored better, and if half or 1 / 4 of the image features are used, the original clothing information is restored slightly worse, but the style retention is better.

[0156] Take the original clothes as real clothes C as an example, and want to output an image of an animated character D wearing clothes of the style of the real clothes C through the virtual dressing model. In the virtual dressing model, the amount of image features used has a significant impact on the effect of the final generated image. Specifically, when all image features are used, the model can restore the detailed information of the original clothes C more accurately, so that the clothes worn by the animated character D in the image are almost completely consistent with the real clothes C. This method ensures a high degree of restoration, but may less integrate the specific style of the animated character D. On the contrary, if you choose to use half or a quarter of the image features, although the degree of restoration of the details of the original clothes C will be slightly reduced, this method can better integrate the style of the animated character D while keeping the basic style of the clothes the same as the real clothes C. Therefore, the clothes worn by the animated character D in the output image are not only similar in style to the real clothes C, but also the overall style is closer to the animation style of the animated character D, which increases the interest and coordination of the image.

[0157] In the process of training the virtual dressing model, due to the complexity of the model and the need for diversity, multiple rounds of training are indispensable. This means that multiple types of data need to be repeatedly input, including the first clothing image, the first text description, and the first posture information. In order to improve the generalization ability and robustness of the model, that is, to enable the model to better adapt to different types of inputs and generate high-quality dressing images that meet user needs, during the training process, the input to the original virtual dressing model can be controlled according to the preset probability to train the data of the target virtual dressing model.

[0158] Specifically, in a possible embodiment, during the process of training the dressing model, fine data control can be performed for each round of the entire training process. During each round of training, the target dressing data input to the original virtual dressing model can be the first clothing image and the first text, or the first clothing image and the first posture, but this method may increase the complexity of model learning due to the frequent changes in data combinations.

[0159] Therefore, in another possible embodiment, the target dressing data inputted in several rounds of the training process can be controlled. This means that data is not discarded in every round, but the input data is adjusted in these specific rounds according to preset rules (such as data control in one round out of every five rounds). On this basis, the target dressing data inputted into the original virtual dressing model can be the first clothing image, the first text and the first posture, or the first clothing image and the first text, or the first clothing image and the first posture. This ensures that the model can be exposed to a variety of data inputs, and avoids training instability caused by frequent changes in data combinations.

[0160] Exemplarily, taking an example with a preset probability of 0.2, in each training, the system has a 20% probability of performing a specific data discarding operation to achieve Classfier Free Guidance. This strategy aims to improve the controllability of the model in the subsequent generation of dressing images. Specifically, when the system decides to discard the first text, the target dressing data of this round will only contain the first clothing image and the first posture, forcing the model to rely more on posture information to generate the dressing effect; similarly, when the first posture is discarded, only the first clothing image and the first text are used as input, prompting the model to rely more on text descriptions. In this way, we can indirectly guide and control the output of the model by adjusting the input data without relying on additional classifiers.

[0161] On this basis, the model parameters of the original virtual dressing model can be adjusted according to the difference between the first dressing image and the first original image, and the difference between the first dressing image and the target dressing image, to obtain the target virtual dressing model, wherein the target dressing image is a dressing image generated according to the input target dressing data.

[0162] The application of the above embodiment ensures that the model can be exposed to complete and diverse data inputs (including the first clothing image, the first text and the first posture) in most rounds, which helps the model to fully learn and understand the correlation between different data dimensions, thereby improving the accuracy and richness of its dressing effect. Secondly, by adjusting the combination of input data in a specific round, such as only inputting the first clothing image and the first text or the first clothing image and the first posture, the model is forced to learn and reason in the absence of some information, which enhances the robustness and adaptability of the model, so that the model can learn to produce reasonable dressing effects in the case of incomplete data or noise, and further avoids the problem of training instability caused by frequent changes in data combinations. In summary, controlling the target dressing data in specific rounds of training not only ensures that the model can be exposed to diverse data inputs, improves its generalization ability, robustness and richness of dressing effects, but also enhances the stability of training by reducing the frequency of data combination changes, laying a solid foundation for finally obtaining a high-performance virtual dressing model.

[0163] Corresponding to the specific steps of inputting the first clothing image, the first text and the first posture into the original virtual dressing model to obtain the first dressing image output by the original virtual dressing model, see Figure 8 , Figure 8 This is a second schematic diagram of the virtual dress-changing model training in an embodiment of the present invention. Figure 8 In the figure, the original virtual dressing model 700 includes an image feature extraction module 801 (including the aforementioned image encoder 8011, a linear layer 8012 and a normalization layer 8013), a text feature extraction module 802, a first diffusion module 803 and a second diffusion module 804, wherein the first diffusion module 803 includes a forward diffusion submodule 8031 ​​and a reverse diffusion submodule 8032, and the second diffusion module 804 includes a forward diffusion submodule 8041 and a reverse diffusion submodule 8042.

[0164] The first clothing image 701 is input into the image feature extraction module 801 to obtain the first image feature 100 .

[0165] The first text 702 is input into the text feature extraction module 802 to obtain the first text feature 200 .

[0166] The first posture 703 is input to the reverse diffusion submodule 8032 of the first diffusion module 803 to obtain the target posture 300 .

[0167] The first image feature 100 and the first text feature 200 are input into the forward diffusion submodule 8041 and the reverse diffusion submodule 8042 of the second diffusion module 804, and the target posture 300 is input into the reverse diffusion submodule 8042 of the second diffusion module 804 to obtain the predicted dressing image output by the second diffusion module 804 as the first dressing image 704 output by the original virtual dressing model 700.

[0168] By applying the above embodiment, by integrating the image feature extraction module and the text feature extraction module, the model can accurately capture the visual details of the clothing and the user's text description intention, ensuring that the dressing effect of the trained virtual dressing model not only conforms to the characteristics of the clothing itself, but also meets the personalized needs of the user. At the same time, the first diffusion module and the second diffusion module introduced, especially the collaborative work of the forward and reverse diffusion submodules, effectively improve the adaptability and naturalness of the posture during the dressing process. The target virtual dressing model finally obtained can generate highly fitting and natural and smooth dressing images according to different text prompts and posture changes of different characters, which not only realizes the accurate replacement of clothing, but also greatly enhances the naturalness, flexibility and realism of dressing, providing users with a richer and more personalized virtual dressing experience.

[0169] In order to improve the performance of the trained virtual dressing model, in a possible embodiment, the second diffusion module further includes a linear projection submodule.

[0170] On this basis, the aforementioned step of inputting the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module may specifically include:

[0171] The first image feature and the first text feature are input into the linear projection submodule to obtain a first linear feature.

[0172] The first linear feature is input to a forward diffusion submodule and a reverse diffusion submodule of a second diffusion module.

[0173] Among them, the linear projection submodule can be a linear projection layer, such as Cross-Attention.

[0174] Corresponding to the specific steps of inputting the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, see Fig. 9 , Fig. 9 This is a third schematic diagram of the virtual dress-changing model training in an embodiment of the present invention. The second diffusion module 804 also includes a linear projection submodule 8043.

[0175] The first image feature 100 and the first text feature 200 are input into the linear projection submodule 8043 to obtain the first linear feature 400 .

[0176] The first linear feature 400 is input to the forward diffusion submodule 8041 and the reverse diffusion submodule 8042 of the second diffusion module 804 .

[0177] Corresponding to the specific process of the virtual dressing model training described above, in S104, according to the difference between the first dressing image and the first original image, the model parameters of the original virtual dressing model can be adjusted in the following two ways:

[0178] Method 1: Adjust the module parameters of the image feature extraction module.

[0179] Method 2: Adjust the module parameters of the image feature extraction module and the linear projection submodule to obtain the target virtual dressing model.

[0180] When adjusting the model parameters of the original virtual dressing model through the above-mentioned method 1, other modules except the image feature extraction module are frozen. When adjusting the model parameters of the original virtual dressing model through the above-mentioned method 2, other modules except the image feature extraction module and the linear projection sub-module are frozen.

[0181] By applying the above method 1, the optimized image feature extraction module can more accurately capture and analyze the key information in the input image, such as clothing texture, color, shape and other details. After the virtual dressing model training is completed, the realism and naturalness of the dressing effect are greatly improved, providing users with a richer and more personalized virtual dressing experience.

[0182] By applying the above method 2, the optimized image feature extraction module can more deeply explore the detailed information in the image, including the material, texture, color gradient, etc. of the clothing, making the model more accurate in capturing and restoring clothing features. The adjustment of the parameters of the linear projection submodule further enhances the cross-modal fusion capability of the model. As a key bridge connecting image and text features, the optimization of this module makes the mapping relationship between image features and description texts closer and more accurate. This not only helps the model better understand the user's intention to change clothes, but also can automatically generate clothing images that match the text description after the virtual dressing model is trained, achieving a more personalized and customized dressing effect. It shows stronger robustness and adaptability when processing complex and changeable input data, and can cope with clothing and human images of different styles, sizes, and lighting conditions, ensuring the stability and consistency of the dressing effect.

[0183] Corresponding to the aforementioned training method for the virtual dressing model, the embodiment of the present invention further provides a virtual dressing method, see Fig.10 , Fig.10 The flowchart of the virtual dressing method in the embodiment of the present invention includes:

[0184] S1001, obtaining data to be replaced.

[0185] The data to be changed into clothes includes a second clothing image, a second text and a second posture of the second clothing.

[0186] S1002, inputting the to-be-dressed data into a target virtual dressing model to obtain a second dressing image output by the target virtual dressing model, wherein the target virtual dressing model is pre-trained using any of the steps of the aforementioned virtual dressing model training method.

[0187] The second character in the second dressing-up image wears the second clothing, the posture of the second character in the second dressing-up image is the second posture, and the second dressing-up image conforms to the description of the second text.

[0188] By applying the above embodiment, by obtaining the data for dressing up that includes the clothing that the user expects to change into (the second clothing image of the second clothing), the specified text (the second text) and the expected posture (the second posture), and directly inputting the introduced pre-trained target virtual dressing model, the model can adjust the presentation of clothing according to different text prompts and different character postures, thereby ensuring the naturalness and fit of the dressing image in different postures, greatly enhancing the flexibility and realism of dressing up, and providing users with a richer and more personalized virtual dressing experience.

[0189] The virtual dressing method can refer to the relevant instructions of the aforementioned S101-S104, which will not be repeated here.

[0190] In order to more clearly illustrate the training method of the virtual dressing model and the virtual dressing method of the embodiment of the present invention, the training method of the virtual dressing model and the virtual dressing method will be exemplified below in combination with the training model and a specific flowchart of using the model to generate dressing images.

[0191] See also Fig.11 , Fig.11 The flowchart of the training and application of the virtual dressing model in the embodiment of the present invention includes:

[0192] S1101, collecting clothing images of characters in a required scene, which is equivalent to obtaining a first original image of a first character as mentioned above;

[0193] S1102, using a posture recognition and depth estimation algorithm to obtain a human posture map and a depth map, which is equivalent to using the posture recognition algorithm to identify the first person in the first original image to obtain human key points as the first posture; or using the depth estimation algorithm to identify the first person in the first original image to obtain a human depth image as the first posture;

[0194] S1103, using a segmentation algorithm in conjunction with a face detection algorithm to segment a clothing region from the image, which is equivalent to searching for a first original image of the first person from the second preset data set, and using a semantic segmentation algorithm to segment the first original image to obtain a first clothing image of the first clothing;

[0195] S1104, preprocessing the image of the clothing area;

[0196] S1105, using a visual encoder such as CLIP to obtain clothing features and using a text description algorithm such as BLIP2 to obtain prompt words, which is equivalent to inputting the first clothing image into an image feature extraction module to obtain the first image features, and obtaining the first text for describing other elements in the first original image except the first clothing;

[0197] S1106, in the process of generating an image by the diffusion model, injecting clothing feature information, which is equivalent to inputting the first clothing image into the original virtual dressing model to obtain the first dressing image output by the original virtual dressing model;

[0198] S1107, combining the prompt words, the target human posture (or the target depth character depth map) and other training models, which is equivalent to the above-mentioned inputting the first clothing image, the first text and the first posture into the original virtual dressing model, obtaining the first dressing image output by the original virtual dressing model, and adjusting the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image, to obtain the target virtual dressing model;

[0199] S1108, after obtaining the model, refer to the clothing and generate a picture according to the specified task posture, which is equivalent to the aforementioned acquisition of the data to be changed, wherein the data to be changed includes a second clothing image of the second clothing, a second text and a second posture; the data to be changed is input into the target virtual dressing model to obtain the second dressing image output by the target virtual dressing model.

[0200] For example, see Fig.12 , Fig.12 1201, the second clothing image 1202, the second text 1202 and the second posture 1203 of the second clothing are input into the target virtual dressing model, and the second dressing image 1205 can be obtained. Image 1204 is the original image of the character in the second dressing image before the dressing.

[0201] Corresponding to the aforementioned training method for a virtual dress-changing model, the present invention further provides a training device for a virtual dress-changing model, see Fig.13 , Fig.13This is a structural diagram of a virtual dress-changing model training device in an embodiment of the present invention, the device includes:

[0202] An acquisition module 1301 is used to acquire a first original image of a first person and acquire a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing;

[0203] A determination module 1302 is used to obtain a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image;

[0204] An input module 1303, configured to input the first clothing image, the first text, and the first posture into an original virtual dressing model, to obtain a first dressing image output by the original virtual dressing model;

[0205] The adjustment module 1304 is configured to adjust the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image to obtain a target virtual dressing model.

[0206] By applying the above embodiment, the clothing image of the clothing worn by the first character in the first original image and the text describing the non-clothing elements are accurately extracted. Since the clothing is attached to the body of the character, the different postures of the character will affect the visual effect of the clothing presented in the image. Therefore, the posture of the first character in the first original image is also accurately captured, and this information is used as input. Furthermore, with the first original image as the true value, by comparing the difference between the dressing image output by the model and the original image, the model parameters are adjusted in a targeted manner to narrow the gap between the dressing image and the original image, so that the target virtual dressing model finally obtained can adjust the presentation of clothing according to different text prompts and different character postures, thereby ensuring the naturalness and fit of the dressing image in different postures, greatly enhancing the flexibility and realism of dressing, and providing users with a richer and more personalized virtual dressing experience.

[0207] In a possible embodiment, the acquisition module includes:

[0208] Obtaining a first submodule, for searching a first original image of a first person and a first clothing image of the first clothing from a first preset data set;

[0209] or,

[0210] Obtain a second submodule, which is used to search for a first original image of a first person from a second preset data set;

[0211] A third submodule is obtained, which is used to segment the first original image using a semantic segmentation algorithm to obtain a first clothing image of the first clothing.

[0212] In a possible embodiment, the determining module includes:

[0213] Determine a first submodule, which is used to determine a second text for describing all elements in the first original image by using a text description algorithm;

[0214] Determine a second submodule for displaying the second text;

[0215] A third submodule is determined, which is used to respond to a deletion instruction for a third text in the second text for describing the first clothing, delete the text indicated by the deletion instruction, and use the remaining text as the first text.

[0216] In a possible embodiment, the determining module includes:

[0217] Determine a fourth submodule, which is used to use a posture recognition algorithm to identify the first person in the first original image to obtain human body key points as a first posture;

[0218] or,

[0219] A fifth submodule is determined, which is used to use a depth estimation algorithm to identify the first person in the first original image to obtain a human body depth image as a first posture.

[0220] In a possible embodiment, the virtual dressing model includes an image feature extraction module, a text feature extraction module, a first diffusion module and a second diffusion module, and the diffusion module includes a forward diffusion submodule and a reverse diffusion submodule;

[0221] The input module comprises:

[0222] An input first submodule is used to input the first clothing image into the image feature extraction module to obtain a first image feature;

[0223] An input second submodule is used to input the first text into the text feature extraction module to obtain a first text feature;

[0224] An input third submodule is used to input the first posture into the reverse diffusion submodule of the first diffusion module to obtain a target posture;

[0225] A fourth submodule is input, which is used to input the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, and input the target posture into the reverse diffusion submodule of the second diffusion module, so as to obtain the predicted dressing image output by the second diffusion module as the first dressing image output by the original virtual dressing model.

[0226] In a possible embodiment, the second diffusion module further includes a linear projection submodule;

[0227] The input fourth submodule comprises:

[0228] An input first subunit is used to input the first image feature and the first text feature into the linear projection submodule to obtain a first linear feature;

[0229] An input second sub-unit, used for inputting the first linear characteristic into a forward diffusion sub-module and a reverse diffusion sub-module of the second diffusion module;

[0230] The adjustment module comprises:

[0231] The first submodule is adjusted to adjust the module parameters of the image feature extraction module, or the module parameters of the image feature extraction module and the linear projection submodule, to obtain a target virtual dressing model.

[0232] In a possible embodiment, the device further includes:

[0233] a screening module, configured to input target dressing data into an original virtual dressing model to obtain a target dressing image output by the original virtual dressing model, wherein the target dressing data includes the first clothing image, the first text, and the first posture, or the first clothing image and the first text, or the first clothing image and the first posture;

[0234] The adjustment module comprises:

[0235] The second adjustment submodule is used to adjust the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image and the difference between the first dressing image and the target dressing image to obtain the target virtual dressing model.

[0236] Corresponding to the aforementioned virtual dressing method, the present invention also provides a virtual dressing device, see Fig.14 , Fig.14 This is a schematic diagram of the structure of a virtual dressing device in an embodiment of the present invention, the device includes:

[0237] The to-be-changed clothing data acquisition module 1401 is used to acquire the to-be-changed clothing data, wherein the to-be-changed clothing data includes a second clothing image, a second text, and a second posture of a second clothing;

[0238] The dressing image output module 1402 is used to input the data to be dressed up into the target virtual dressing model to obtain a second dressing image output by the target virtual dressing model, wherein the target virtual dressing model is pre-trained using the steps of any of the aforementioned virtual dressing model training methods, wherein in the second dressing image, the second character wears second clothing, the posture of the second character in the second dressing image is a second posture, and the second dressing image conforms to the description of the second text.

[0239] By applying the above embodiment, by obtaining the data for changing clothes including the clothes that the user expects to change into (the second clothing image of the second clothing), the specified text (the second text) and the expected posture (the second posture), and directly inputting the introduced pre-trained target virtual dressing model, the model can generate highly fitting and natural and smooth dressing images according to different text prompts and posture changes of different characters, which not only realizes the accurate replacement of clothes, but also greatly enhances the naturalness, flexibility and realism of the dressing, providing users with a richer and more personalized virtual dressing experience.

[0240] The embodiment of the present invention further provides an electronic device, such as Fig.15 As shown, it includes a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0241] Memory 1503, used for storing computer programs;

[0242] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0243] Acquire a first original image of a first person and acquire a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing;

[0244] Acquire a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image;

[0245] Inputting the first clothing image, the first text, and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model;

[0246] According to the difference between the first dress-changing image and the first original image, the model parameters of the original virtual dress-changing model are adjusted to obtain a target virtual dress-changing model.

[0247] Or, obtaining the data for clothing to be changed, wherein the data for clothing to be changed includes a second clothing image, a second text, and a second posture of the second clothing;

[0248] The to-be-dressed data is input into a target virtual dressing model to obtain a second dressing image output by the target virtual dressing model, wherein the target virtual dressing model is trained in advance by any of the aforementioned virtual dressing model training method steps.

[0249] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0250] The communication interface is used for communication between the above terminal and other devices.

[0251] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0252] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0253] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the training method of the virtual dressing model or the virtual dressing method described in any of the above embodiments is implemented.

[0254] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the virtual dressing model training method or virtual dressing method described in any of the above embodiments.

[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0256] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0257] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0258] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A training method for a virtual dress-up model, characterized in that: The method comprises: Acquire a first original image of a first person and acquire a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing; Acquire a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image; Inputting the first clothing image, the first text, and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model; According to the difference between the first dress-changing image and the first original image, the model parameters of the original virtual dress-changing model are adjusted to obtain a target virtual dress-changing model.

2. The method according to claim 1, characterized in that The step of acquiring a first original image of a first person and acquiring a first clothing image of a first clothing includes: Searching for a first original image of a first person and a first clothing image of the first clothing from a first preset data set; or, Searching for a first original image of a first person from a second preset data set; The first original image is segmented using a semantic segmentation algorithm to obtain a first clothing image of the first clothing.

3. The method according to claim 1, characterized in that The step of obtaining a first text for describing other elements in the first original image except the first clothing includes: Determine a second text for describing all elements in the first original image by using a text description algorithm; displaying the second text; In response to a deletion instruction for the third text in the second text for describing the first clothing, the text indicated by the deletion instruction is deleted, and the remaining text is used as the first text.

4. The method according to claim 1, characterized in that: The obtaining of a first posture of the first person in the first original image includes: Using a posture recognition algorithm to identify the first person in the first original image to obtain key points of the human body as a first posture; or, A first person in the first original image is identified by using a depth estimation algorithm to obtain a human body depth image as a first posture.

5. The method according to claim 1, characterized in that The virtual dressing model includes an image feature extraction module, a text feature extraction module, a first diffusion module and a second diffusion module, wherein the diffusion module includes a forward diffusion submodule and a reverse diffusion submodule; The step of inputting the first clothing image, the first text, and the first posture into an original virtual dressing model to obtain a first dressing image output by the original virtual dressing model includes: Inputting the first clothing image into the image feature extraction module to obtain a first image feature; Inputting the first text into the text feature extraction module to obtain a first text feature; Inputting the first posture into the reverse diffusion submodule of the first diffusion module to obtain a target posture; The first image feature and the first text feature are input into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module, and the target posture is input into the reverse diffusion submodule of the second diffusion module to obtain the predicted dressing image output by the second diffusion module as the first dressing image output by the original virtual dressing model.

6. The method according to claim 5, characterized in that The second diffusion module also includes a linear projection submodule; The step of inputting the first image feature and the first text feature into the forward diffusion submodule and the reverse diffusion submodule of the second diffusion module includes: Inputting the first image feature and the first text feature into the linear projection submodule to obtain a first linear feature; inputting the first linear feature into a forward diffusion submodule and a reverse diffusion submodule of the second diffusion module; The adjusting the model parameters of the original virtual dressing model to obtain the target virtual dressing model includes: The module parameters of the image feature extraction module, or the module parameters of the image feature extraction module and the linear projection submodule are adjusted to obtain a target virtual dressing model.

7. The method according to claim 1, characterized in that The method further comprises: Inputting target dressing data into an original virtual dressing model to obtain a target dressing image output by the original virtual dressing model, wherein the target dressing data includes the first clothing image, the first text, and the first posture, or the first clothing image and the first text, or the first clothing image and the first posture; The step of adjusting the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image to obtain the target virtual dressing model includes: According to the difference between the first dress-changing image and the first original image, and the difference between the first dress-changing image and the target dress-changing image, the model parameters of the original virtual dress-changing model are adjusted to obtain the target virtual dress-changing model.

8. A virtual dressing method, characterized in that: The method comprises: Acquiring data for clothing to be changed, wherein the data for clothing to be changed includes a second clothing image, a second text, and a second posture of the second clothing; The data to be changed into is input into a target virtual changing model, and a second changing image is output by the target virtual changing model, wherein the target virtual changing model is pre-trained using the method steps described in any one of claims 1 to 7, wherein in the second changing image, a second character wears the second clothing, a posture of the second character in the second changing image is the second posture, and the second changing image conforms to the description of the second text.

9. A training device for a virtual dress-up model, characterized in that: The device comprises: An acquisition module, configured to acquire a first original image of a first person and a first clothing image of a first clothing, wherein the first person in the first original image wears the first clothing; a determination module, configured to obtain a first text for describing other elements in the first original image except the first clothing, and a first posture of the first person in the first original image; An input module, used for inputting the first clothing image, the first text and the first posture into an original virtual dressing model, to obtain a first dressing image output by the original virtual dressing model; The adjustment module is used to adjust the model parameters of the original virtual dressing model according to the difference between the first dressing image and the first original image to obtain a target virtual dressing model.

10. A virtual dressing device, characterized in that: The device comprises: a module for acquiring data for clothing to be changed, used for acquiring data for clothing to be changed, wherein the data for clothing to be changed includes a second clothing image, a second text and a second posture of the second clothing; A dressing image output module is used to input the data to be dressed into a target virtual dressing model to obtain a second dressing image output by the target virtual dressing model, wherein the target virtual dressing model is pre-trained using the method steps described in any one of claims 1 to 7, wherein a second character in the second dressing image wears the second clothing, a posture of the second character in the second dressing image is the second posture, and the second dressing image conforms to the description of the second text.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-7 or 8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1-7 or 8 are implemented.

Citation Information

Cited By

  • Controllable specific figure portrait generation method and device, storage medium and equipment

    CN118447110A