Virtual try-on method and apparatus, and storage medium

Abstract

A virtual try-on method and apparatus, and a storage medium. The method comprises: acquiring a clothing image to be subjected to try-on and a first model image; on the basis of the clothing image to be subjected to try-on, determining a target clothing image; on the basis of the first model image, determining a try-on area image and a first key point, wherein a try-on area comprises an area of model clothing, a body area which is in contact with the model clothing, and a background area which is in contact with the model clothing; and on the basis of the target clothing image, the try-on area image and the first key point, generating a try-on effect image by means of a trained diffusion model.

Description

Virtual fitting method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed on February 21, 2024, with application number 202410191702.6 and invention name “A virtual fitting method, device and storage medium”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] The present application relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a virtual fitting method, device, and storage medium. Background Art

[0003] Virtual fitting technology is of great value in e-commerce, online shopping and other fields. On the one hand, it can better display the seller's products, and on the other hand, it can allow consumers to experience the visual effects of wearing them before purchasing.

[0004] In some technical solutions, the fitting effect images generated by the virtual fitting technology, such as the lighting, shadows, wrinkles, texture, etc. of the clothes, may be significantly different from the actual images, resulting in low quality of the generated fitting effect images. Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] In this embodiment, a virtual fitting method, device, and storage medium are provided.

[0007] In a first aspect, a virtual fitting method is provided in this embodiment, the method comprising:

[0008] Obtaining pictures of clothing to be tried and pictures of the first model;

[0009] Determining a target clothing image according to the clothing image to be tried;

[0010] Determining a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the model's limbs in contact with the model's clothing, and a background area in contact with the model's clothing; the model's clothing is determined based on the type of the target clothing;

[0011] Generate a fitting effect image based on the target clothing image, the fitting area image, and the first key point using the trained diffusion model;

[0012] An iterative cycle of the iterative prediction includes: inputting the target clothing image, the fitting area image, the first key point and the noise image of the current iterative cycle into the trained diffusion model to generate a target fitting image of the current iterative cycle; using the target fitting image of the current iterative cycle as the noise image of the next iterative cycle; using the target fitting image of the iterative cutoff cycle as the fitting effect image; wherein the noise image of the first iterative cycle is a random noise image.

[0013] In some embodiments, determining the target clothing image based on the clothing image to be tried on includes:

[0014] The clothing image to be tried on is classified according to a preset classifier to obtain a first clothing image or a second clothing image.

[0015] In some embodiments, determining the target clothing image based on the clothing image to be tried on includes:

[0016] The first clothing image is segmented according to a preset first clothing segmentation model to obtain the target clothing image.

[0017] In some embodiments, determining the target clothing image based on the clothing image to be tried on includes:

[0018] The second clothing image is segmented according to a preset second clothing segmentation model to obtain the target clothing image.

[0019] In some embodiments, the virtual fitting method further includes:

[0020] A second key point is determined according to the target clothing image.

[0021] In some embodiments, generating a fitting effect image based on the target clothing image, the fitting area image, and the first key point using a trained diffusion model includes:

[0022] The fitting effect diagram is generated according to the target clothing diagram, the second key points, the fitting area diagram, the first key points and the trained diffusion model.

[0023] In some embodiments, training the diffusion model includes:

[0024] Add random noise to the training model image to obtain the training image set;

[0025] The diffusion model is trained according to the training image set until the diffusion model converges, thereby obtaining the trained diffusion model.

[0026] In a second aspect, a virtual fitting device is provided in this embodiment, comprising:

[0027] An acquisition module is configured to acquire a clothing image to be tried on and a first model image;

[0028] A first determining module is configured to determine a target clothing image based on the clothing image to be tried on;

[0029] A second determining module is configured to determine a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the limbs in contact with the model's clothing, and a background area in contact with the model's clothing; the model's clothing is determined based on a type of target clothing;

[0030] a generation module configured to generate a fitting effect image based on the target clothing image, the fitting area image, and the first key point using a trained diffusion model;

[0031] An iterative cycle of the iterative prediction includes: inputting the target clothing image, the fitting area image, the first key point and the noise image of the current iterative cycle into the trained diffusion model to generate a target fitting image of the current iterative cycle; using the target fitting image of the current iterative cycle as the noise image of the next iterative cycle; using the target fitting image of the iterative cutoff cycle as the fitting effect image; wherein the noise image of the first iterative cycle is a random noise image.

[0032] In a third aspect, a computer-readable storage medium is provided in this embodiment, on which a computer program is stored. When the computer program is executed by a processor, the steps of the virtual fitting method described in the first aspect are implemented.

[0033] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0034] Summary of the Figures

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] FIG1 is a hardware structure block diagram of a terminal for executing a virtual fitting method according to an embodiment of the present application;

[0037] FIG2 is a flow chart of a virtual fitting method according to an embodiment of the present application;

[0038] FIG3 is a flow chart of an iterative cycle of an embodiment of the present application;

[0039] FIG4 is a flowchart of another virtual fitting method according to an embodiment of the present application;

[0040] FIG5 is a schematic diagram of generating a fitting effect diagram according to an embodiment of the present application;

[0041] FIG6 is a schematic diagram of another method for generating a fitting effect diagram according to an embodiment of the present application;

[0042] FIG7 is a structural block diagram of a virtual fitting device according to an embodiment of the present application.

[0043] Details

[0044] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0045] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0046] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, when running on a terminal, Figure 1 is a hardware structure block diagram of a terminal that executes a virtual fitting method of an embodiment of the present application. As shown in Figure 1, the terminal may include one or more (only one is shown in Figure 1) processors 102 and a memory 104 configured to store data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above-mentioned terminal may include a transmission device 106 and an input and output device 108 with communication functions. It will be understood by those skilled in the art that the structure shown in Figure 1 is only illustrative and does not limit the structure of the above-mentioned terminal. For example, the terminal may also include more or fewer components than those shown in Figure 1, or have a different configuration than that shown in Figure 1.

[0047] The memory 104 can be configured to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a virtual fitting method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is configured to receive or transmit data via a network. The network may include a wireless network provided by the terminal's telecommunications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0049] This embodiment provides a virtual fitting method. FIG2 is a flow chart of a virtual fitting method according to an embodiment of the present application. As shown in FIG2 , the flow chart includes the following steps S210-S240:

[0050] Step S210: obtaining a clothing image to be tried on and a first model image.

[0051] For example, the first model image includes a first target model. The clothing image to be tried on includes target clothing to be tried on or changed into by the first target model. The target clothing may include tops, pants, skirts, shoes, and accessories, etc., without specific limitation herein.

[0052] Step S220: determining a target clothing image according to the clothing image to be tried on.

[0053] For example, based on the acquired clothing image to be tried on, a target clothing image of the target clothing item to be tried on or changed by the first target model in the clothing image to be tried on is determined. Here, the clothing image to be tried on can be identified or classified using an image recognition algorithm or classifier to determine the target clothing image.

[0054] Step S230, determining a fitting area map and a first key point based on the first model image; wherein the fitting area includes the area of ​​the model's clothing, the limb area in contact with the model's clothing, and the background area in contact with the model's clothing; the model's clothing is determined according to the type of the target clothing.

[0055] Exemplarily, a try-on area diagram and a first key point are determined based on a first model diagram. The first key point here can be a skeletal point (pose) of the first target model. Exemplarily, the first key point is predicted using a pre-trained skeletal point prediction model. The try-on area includes the area of ​​the model's clothing, the limb area in contact with the model's clothing, and the background area in contact with the model's clothing. The model's clothing can be determined based on the type of the target clothing. For example, if the target clothing is a top, the model's clothing is determined to be a top, and the corresponding try-on area includes the area of ​​the top, the limb area in contact with the top, and the background area in contact with the top.

[0056] For example, a mask image of the fitting area in the first model image may be obtained first, and then the fitting area portion in the first model image may be removed based on the first model image and the mask image of the fitting area to obtain a fitting area image.

[0057] Step S240 : generating a fitting effect diagram based on the target clothing diagram, the fitting area diagram, and the first key point using the trained diffusion model.

[0058] The virtual fitting or virtual dressing in this embodiment refers to a given target model and target clothing, and the target model is made to wear the target clothing through the virtual fitting method provided in this embodiment.

[0059] In this embodiment, a high-quality fitting effect image is generated based on the target clothing image, the fitting area image of the first model image, and the first key point of the first target model through the trained diffusion model.

[0060] In some embodiments, a fitting effect image is generated based on a target clothing image, a fitting area image and a first key point through a trained diffusion model, including: inputting the target clothing image, the fitting area image, the first key point and the noise image into the trained diffusion model; the trained diffusion model generates the fitting effect image through iterative prediction.

[0061] In this embodiment, by inputting the target clothing image, the fitting area image, the first key point and the noise image into the trained diffusion model, the trained diffusion model generates the fitting effect image through iterative prediction, which can improve the quality of the fitting effect image.

[0062] In some embodiments, as shown in FIG3 , an iterative cycle of iterative prediction includes the following steps:

[0063] Step S310 : Input the target clothing image, the fitting area image, the first key point, and the noise image of the current iteration cycle into the trained diffusion model to generate the target fitting image of the current iteration cycle.

[0064] Step S320 , using the target fitting image of the current iteration cycle as the noise image of the next iteration cycle; wherein the noise image of the first iteration cycle is a random noise image.

[0065] Step S330: Using the target fitting image at the end of the iteration period as the fitting effect image.

[0066] In some embodiments, determining a target clothing image based on a clothing image to be tried on includes:

[0067] The clothing image to be tried on is classified according to a preset classifier to obtain a first clothing image or a second clothing image.

[0068] Exemplarily, the clothing image to be tried on includes target clothing, and the clothing image to be tried on is classified according to a preset classifier to obtain a second clothing image or a second model image.

[0069] In some embodiments, determining a target clothing image based on a clothing image to be tried on includes:

[0070] The first clothing image is segmented according to a preset first clothing segmentation model to obtain a target clothing image.

[0071] For example, the first clothing image here can be a clothing image with a complex background, where the complex background can be a colored background or other patterned background. The first clothing image is segmented using a preset first clothing segmentation model to obtain a target clothing image with a specific background, where the specific background can be a solid color background.

[0072] In this embodiment, the first clothing image is preprocessed to obtain a target clothing image with a specific background, and the target clothing image with a specific background is input into the trained diffusion model to generate a corresponding fitting effect image, thereby improving the prediction efficiency of the diffusion model.

[0073] In some embodiments, determining a target clothing image based on a clothing image to be tried on includes:

[0074] The second clothing image is segmented according to a preset second clothing segmentation model to obtain a target clothing image.

[0075] For example, the second clothing image may include a second target model, the second target model wearing the target clothing, and the second clothing image is segmented using a preset second clothing segmentation model to obtain a target clothing image. The first target model and the second target model may be the same model or different models. When the first target model and the second target model are the same model, the first model image and the second model image may be images of the same model wearing different clothing. When the first target model and the second target model are different models, the first model image and the second model image may be images of different models wearing different clothing.

[0076] In this embodiment, the second clothing image is segmented according to a preset second clothing segmentation model to obtain a target clothing image, thereby achieving clothing change between two target models.

[0077] In some embodiments, the virtual fitting method further includes:

[0078] Determining the second key point based on the target clothing image. In some embodiments, generating a fitting effect image based on the target clothing image, the fitting area image, and the first key point using a trained diffusion model includes: generating the fitting effect image based on the target clothing image, the second key point, the fitting area image, the first key point, and the trained diffusion model.

[0079] In some embodiments, training the diffusion model includes: adding random noise to a training model image to obtain a training image set; and training the diffusion model based on the training image set until the diffusion model converges to obtain a trained diffusion model.

[0080] This embodiment also provides a virtual fitting method. FIG4 is a flow chart of another virtual fitting method according to an embodiment of the present application. As shown in FIG4 , the flow chart includes the following steps S410-S450:

[0081] Step S410: prepare a training image set.

[0082] Exemplarily, step S410 includes:

[0083] Prepare training model images;

[0084] Add random noise to the training model image to obtain the training image set.

[0085] Step S420 : training the diffusion model according to the training image set to obtain a trained diffusion model.

[0086] Exemplarily, step S420 includes:

[0087] The training image set is input into the diffusion model, and the diffusion model is trained until the diffusion model converges to obtain a trained diffusion model.

[0088] In some embodiments, the model can be trained for a fixed number of iterations, such as 500,000 iterations. The loss function is the Mean Sequential Error (MSE), which predicts the mean and variance of the noise. When the model training reaches the fixed number of iterations, the trained diffusion model is output.

[0089] During the training process of the diffusion model, the output of the diffusion model can be a noise-free training model image, or it can be random noise itself; the purpose of training is to enable the diffusion model to distinguish the noise part in the image, thereby predicting a noise-free image.

[0090] The diffusion model starts with a training model image and gradually adds Gaussian noise through a diffusion process. In theory, this is not done all at once, but rather proceeds step by step through a series of discrete time steps, where noise is further added at each step until the data becomes a pure noise state. The "reparameterization trick" is used to simplify the calculation of noise addition, allowing the model to directly generate noisy data at any time step. The subsequent inverse diffusion process (i.e., the image generation process described below) starts from the noisy state, gradually reduces the noise and restores the data. The model learns to reverse the effects of noise at each iterative step. The reparameterization technique here helps to optimize the denoising process and improve the denoising efficiency of the model at each time step. This iterative approach allows the diffusion model to efficiently process and generate complex image data.

[0091] S430: Obtain a clothing image of the clothes to be tried on and a first model image.

[0092] Exemplarily, the original clothing image is first obtained and sent as input to a clothing classifier to determine whether the clothing is supported by the virtual fitting algorithm. If so, the original clothing image is used as the clothing image to be tried on; if not, an error prompt is given to the user. The clothing classifier determines whether it is a supported clothing, and the main requirements include: single piece of clothing, flat (non-folded), front, appropriate size ratio, etc. The input of the clothing classifier is the clothing image, and the output of the clothing classifier is whether it is a clothing supported by the virtual fitting method (yes or no). The clothing image to be tried on can be a first clothing image or a second clothing image, wherein the first clothing image only contains the target clothing, and the second clothing image contains a second target model wearing the target clothing. In some exemplary embodiments, the clothing image to be tried on can be classified according to a preset classifier to determine whether the clothing image to be tried on is the first clothing image or the second clothing image.

[0093] Exemplarily, the first target model in the first model image can be a preset model or a model in an image uploaded by a user. If the image is uploaded by a user, a classifier can be used to determine whether the image meets the requirements. The main classification criteria of the classifier include: appropriate portrait proportions, front view, posture that meets clothing model standards, relatively front view (the visible area of ​​the clothing is the front view), and clothing type that meets the requirements. If the requirements are met, the model can be used as the first target model. After selecting the first target model, a target model image is determined. Here, the corresponding image of the selected first target model can be used as the target model image. The target model image is then pre-processed, including background processing. During background processing, if the image background is mainly white (slight lighting changes are allowed), the target model image is directly used as the first model image. If the image background does not meet the above requirements, a portrait segmentation algorithm is used to extract the target model and then synthesize it onto a white background to obtain the first model image.

[0094] Step S440 , determining a target clothing image, a try-on area image, and a first key point according to the clothing image to be tried on and the first model image.

[0095] Exemplarily, obtaining a target clothing image according to a clothing image to be tried on includes:

[0096] In some embodiments, if the clothing image to be tried on is a first clothing image, the following operations are performed on the first clothing image:

[0097] If the background of the first clothing image is mainly white (slight lighting changes are allowed), the first clothing image can be directly used as the target clothing image.

[0098] If the background of the first clothing image is not the above background, the first clothing segmentation model can be used to segment the target clothing and synthesize it onto the white background to obtain the target clothing image.

[0099] If the background of the first clothing image is not the above background, the lining segmentation model can be used to segment the visible lining (neckline, cuffs, hem) so that different perturbations can be made during training to simultaneously support flat or hanging shots, hanger shots, and clothing worn on a mannequin or a real person; in some embodiments, a training clothing image can be divided into three areas: background, clothing body, and lining.

[0100] Exemplarily, a convolutional neural network (CNN) or Transformer can be used to implement the lining segmentation model, wherein the convolutional neural network is generally implemented using U-Net, which performs downsampling and upsampling of the image through symmetrical convolution and deconvolution layers to achieve accurate pixel-level segmentation. The implementation method is to collect data, label, train, and infer (predict). The loss function of the lining segmentation model mainly includes cross entropy loss, Dice loss, etc. The input of the lining segmentation model is the training clothing image, and the output of the lining segmentation model is the clothing mask, the lining mask (including the collar, cuffs, hem, etc. that may reveal the clothing lining), and the background (i.e., the area other than the clothing mask and the lining mask). In other words, the output of the lining segmentation model is a pure clothing image.

[0101] In some embodiments, if the clothing image to be tried on is a second clothing image, the following operations are performed on the second clothing image:

[0102] The target clothing worn by the model is segmented using the second clothing segmentation model to obtain a target clothing image.

[0103] In some embodiments, the target clothing area may be cropped to a suitable size and ratio so that the target clothing fills the image area as much as possible, thereby making fuller use of effective information.

[0104] Exemplarily, determining the fitting area map and the first key point based on the first model image includes:

[0105] In some embodiments, a pre-trained skeletal point (pose) model is used to predict the skeletal point (pose) of the model in the first model image, that is, the first key point. Exemplarily, a convolutional neural network (CNN) or a Transformer can be used to implement the skeletal point model, and the image is downsampled and upsampled through symmetrical convolution and deconvolution layers to achieve accurate pixel-level segmentation. The implementation method is to collect data, label, train, and infer (predict). The loss function includes the mean square error loss (MSE), which measures the square difference between the predicted key point position and the actual key point position, but is sensitive to outliers; the mean absolute error loss (MAE), which measures the absolute value of the difference between the predicted value and the actual value and is less sensitive to outliers; the smooth L1 loss, which combines the advantages of MSE and MAE and is applicable to errors of different sizes; and the structured loss function, which takes into account the relative position and structural relationship between the key points of the human body. In addition, the heat map loss is also widely used in the pose estimation method. It is optimized by calculating the MSE or cross entropy loss between the predicted heat map and the actual heat map. The skeleton point model takes the original training model image as input and outputs skeleton key points. For example, the main skeleton key points of the human body include multiple parts of the head, upper body, and lower body. In the head area, it detects the neck and top of the head. The key points of the upper body include the left and right shoulders, the left upper arm, right upper arm, left forearm, and right forearm, and the left and right waist of the chest and waist. The lower body includes the left and right thighs, the left and right calves, the left and right knees, and the left and right ankles.

[0106] In some embodiments, a try-on area segmentation model can be used to predict a mask image of the try-on area. The try-on area includes a clothing area (clothing of the same category as the target clothing image, for example, if the target clothing image is a top, then the clothing area is also a top area; if the training clothing image is pants, then the clothing area is also pants), a skin area in contact with the clothing area (such as the neck and arms), and part of the background area in contact with the clothing area (because the clothing area after changing clothes may expand more than the original clothing, it is necessary to ensure that the try-on area is covered). The mask image of the try-on area can also be obtained by combining multiple algorithm models, such as human parsing, dense pose estimation, etc. Exemplarily, a convolutional neural network (CNN) or a Transformer can be used to implement the try-on area segmentation model, wherein the convolutional neural network is generally implemented using U-Net, and the image is downsampled and upsampled through symmetrical convolution and deconvolution layers to achieve accurate pixel-level segmentation. The implementation method is to collect data, label, train, and infer (predict). The loss function of the try-on area segmentation model mainly includes cross entropy loss, Dice loss, etc. The input of the fitting area segmentation model is the first model image, and the output of the fitting area segmentation model is the mask image of the fitting area. According to the first model image and the mask image of the fitting area, the fitting area part in the first model image is removed to obtain the fitting area map.

[0107] In some embodiments, a pre-trained skeleton point (pose) model may be used to predict the skeleton points (pose) of the target clothing in the target clothing image and use them as the second key points.

[0108] Step S450 : generating a fitting effect diagram based on the target clothing diagram, the fitting area diagram, and the first key point using the trained diffusion model.

[0109] Exemplarily, the try-on area map, the first key point, the target clothing image, and the noise map are input into a trained diffusion model, which then generates a fitting effect image through iterative prediction. An iterative prediction cycle includes: inputting the try-on area map, the first key point, the target clothing image, and the noise map of the current iteration cycle into the trained diffusion model to generate a target fitting image of the current iteration cycle; using the target fitting image of the current iteration cycle as the noise map of the next iteration cycle; and using the target fitting image of the final iteration cycle as the fitting effect image. The noise map of the first iteration cycle is a random noise map.

[0110] In some embodiments, a schematic flow chart for generating a fitting effect image is shown in FIG5 , where FIG5 takes the first garment image as an example. In the first iteration cycle, the random noise image, the generative region map, the first key point, and the target garment image are input into the trained diffusion model to obtain target fitting image 1. In the second iteration cycle, the target fitting image 1 is input into the trained diffusion model as the noise image, the generative region map, the first key point, and the target garment image, to obtain target fitting image 2. In the nth iteration cycle, the target fitting image n-1 is input into the trained diffusion model as the noise image, the generative region map, the first key point, and the target garment image, to obtain target fitting image n. This continues until the end of the iteration cycle, at which point the output target fitting image is used as the fitting effect image.

[0111] In some embodiments, a schematic flowchart of generating a fitting effect diagram is shown in FIG6 , in which FIG6 takes the clothing diagram to be fitted as the second clothing diagram as an example.

[0112] In some embodiments, the fitting effect image is input into a super-resolution module, and a fitting effect image with a higher resolution is output.

[0113] In some embodiments, a pre-trained portrait segmentation model is used to segment the human body in the fitting image and composite the segmented portion onto the target background to generate a fitting image with a more complex background. For example, if the background of the first model image is non-white, the segmented portion can be composited onto the original background to ensure consistency. Alternatively, a diffusion model can be used to generate the image, making the result more realistic and natural.

[0114] In some embodiments, the try-on area map, the first key point, the second key point, the target clothing map and the noise map can be input into the trained diffusion model. The trained diffusion model generates a fitting effect map through iterative prediction, thereby further improving the quality of the generated fitting effect map.

[0115] The principle behind generating fitting images using the trained diffusion model is as follows: the fitting area map, the first keypoint, and the target garment image undergo a series of encoding steps (where the resolution is gradually reduced and the features are gradually upgraded from low to high dimensions), followed by a series of decoding steps (where the resolution is gradually increased, ultimately restoring the original resolution). At the stage with the lowest resolution and highest feature dimensionality, a cross-attention mechanism is used to fully integrate the high-dimensional features generated by the input encodings, thereby better preserving the details of the target garment image. This process can proceed through multiple stages, for example, generating a low-resolution intermediate result and then generating a higher-resolution result based on this intermediate result. The trained diffusion model begins with an input encoding stage, consisting of multiple convolutional layers. These layers process the fitting area map, the first keypoint, and the target garment image, gradually reducing the image resolution while increasing the feature dimensionality to capture higher-level abstract features. At lower-resolution, deeper layers, self-attention and cross-attention are used to fuse features, ensuring consistency in style and detail in the generated image. The decoding and resolution enhancement phase then proceeds, gradually increasing the image resolution through multiple upsampling layers or similar structures, reconstructing image detail and quality, gradually approaching the original resolution. This intermediate processing stage may generate low-resolution intermediate images, which serve as a transition to produce a higher-resolution final output based on these intermediate results. In this embodiment, the fitable area map, the first key points, and the first garment image are input into a trained diffusion model to generate a high-quality fitting effect image.

[0116] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0117] This embodiment also provides a virtual fitting device that is configured to implement the above-mentioned embodiments and exemplary implementations. Details already described will not be repeated. The terms "module," "unit," "subunit," and the like used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are implemented using software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0118] FIG7 is a block diagram of a virtual fitting device according to an embodiment of the present application. As shown in FIG7 , the device includes:

[0119] An acquisition module 710 is configured to acquire a clothing image to be tried on and a first model image;

[0120] The first determining module 720 is configured to determine a target clothing image based on the clothing image to be tried on;

[0121] The second determining module 730 is configured to determine a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the body in contact with the model's clothing, and a background area in contact with the model's clothing;

[0122] The generation module 740 is configured to generate a fitting effect image based on the target clothing image, the fitting area image and the first key point through the trained diffusion model.

[0123] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented through software or hardware. For modules implemented through hardware, each of the above modules can be located in the same processor; or each of the above modules can be located in different processors in any combination.

[0124] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0125] In one implementation of this embodiment, the electronic device may include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0126] In one implementation of this embodiment, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0127] S1, obtaining a clothing image to be tried on and a first model image;

[0128] S2, determining the target clothing image according to the clothing image to be tried;

[0129] S3, determining a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the body in contact with the model's clothing, and a background area in contact with the model's clothing;

[0130] S4, based on the target clothing image, the fitting area image and the first key point, a fitting effect image is generated through the trained diffusion model.

[0131] It should be noted that the examples in this embodiment can refer to the examples described in the above embodiments and implementation methods, and will not be repeated in this embodiment.

[0132] In addition, in conjunction with a virtual fitting method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the virtual fitting method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the virtual fitting methods in the above embodiment is implemented.

[0133] It should be understood that the embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0134] The accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art may be able to apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understood that although the work involved in the development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered as insufficient disclosure of the present application.

[0135] The term "embodiment" as used in this application refers to features, structures, or characteristics described in connection with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it imply that the embodiments are mutually exclusive and independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments unless there is a conflict.

[0136] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

[0137] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A virtual fitting method, comprising: Obtaining pictures of clothing to be tried on and pictures of the first model; Determining a target clothing image according to the clothing image to be tried; Determining a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the body in contact with the model's clothing, and a background area in contact with the model's clothing; the model's clothing is determined based on the type of the target clothing; Inputting the target clothing image, the fitting area image, the first key points, and the noise image into a trained diffusion model, wherein the trained diffusion model generates a fitting effect image through iterative prediction; An iterative cycle of the iterative prediction includes: inputting the target clothing image, the fitting area image, the first key point and the noise image of the current iterative cycle into the trained diffusion model to generate a target fitting image of the current iterative cycle; using the target fitting image of the current iterative cycle as the noise image of the next iterative cycle; using the target fitting image of the iterative cutoff cycle as the fitting effect image; wherein the noise image of the first iterative cycle is a random noise image.

2. The virtual fitting method according to claim 1, wherein: The step of determining a target clothing image according to the clothing image to be tried on comprises: The clothing image to be tried on is classified according to a preset classifier to obtain a first clothing image or a second clothing image.

3. The virtual fitting method according to claim 2, wherein: The step of determining a target clothing image according to the clothing image to be tried on comprises: The first clothing image is segmented according to a preset first clothing segmentation model to obtain the target clothing image.

4. The virtual fitting method according to claim 2, wherein: The step of determining a target clothing image according to the clothing image to be tried on comprises: The second clothing image is segmented according to a preset second clothing segmentation model to obtain the target clothing image.

5. The virtual fitting method according to claim 1, further comprising: A second key point is determined according to the target clothing image.

6. The virtual fitting method according to claim 5, wherein: The step of inputting the target clothing image, the fitting area image, the first key point, and the noise image into a trained diffusion model, and generating a fitting effect image through iterative prediction by the trained diffusion model, includes: The target clothing image, the second key points, the fitting area image, the first key points and the noise image are input into a trained diffusion model, and the trained diffusion model generates a fitting effect image through iterative prediction.

7. The virtual fitting method according to claim 1, wherein: Training the diffusion model includes: Add random noise to the training model image to obtain the training image set; The diffusion model is trained according to the training image set until the diffusion model converges, thereby obtaining the trained diffusion model.

8. A virtual fitting device, comprising: An acquisition module is configured to acquire a clothing image to be tried on and a first model image; A first determining module is configured to determine a target clothing image based on the clothing image to be tried on; a second determining module configured to determine a fitting area map and a first key point based on the first model image; wherein the fitting area includes an area of ​​the model's clothing, an area of ​​the body in contact with the model's clothing, and a background area in contact with the model's clothing; the model's clothing is determined based on a type of target clothing; a generation module configured to input the target clothing image, the fitting area image, the first key point, and the noise image into a trained diffusion model, wherein the trained diffusion model generates a fitting effect image through iterative prediction; An iterative cycle of the iterative prediction includes: inputting the target clothing image, the fitting area image, the first key point and the noise image of the current iterative cycle into the trained diffusion model to generate a target fitting image of the current iterative cycle; using the target fitting image of the current iterative cycle as the noise image of the next iterative cycle; using the target fitting image of the iterative cutoff cycle as the fitting effect image; wherein the noise image of the first iterative cycle is a random noise image.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the virtual fitting method according to any one of claims 1 to 7 are implemented.

Images