AI technology-based clothing-driven image multi-region feature fusion system

Abstract

The invention provides an AI technology-based clothing-driven image multi-region feature fusion system, and relates to the technical field of image generation. According to the system, an AI technology is combined, an SAM image segmentation model is firstly applied, and a clothing part of an image is extracted; then, a CLIP model is adopted to search a garment image with high similarity with the fashion trend chart from a style library; and then, through the improved Multifunction image generation model, generating a clothing product image with high quality and high fusion degree. An image cutting and splicing processing layer is added in a Unet denoising network in a Multifunction image generation model and is used for enhancing the understanding of the model on clothing image features and eliminating the influence of a background part on a generation region in the early stage of a sampling process. And finally, the background image is converted into the original background image in the later stage of the image sampling process, and a high-precision and coherent garment product image is generated.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and more specifically, to a clothing-driven image multi-region feature fusion system based on AI technology. Background Art

[0002] Although the Stable Diffusion model provides a more generalized content generation capability for the development of automatic content generation by artificial intelligence technology, there are still some challenges in practical applications. For example, clothing product images need to be automatically generated in the user-specified image area without changing the characteristics of clothing structure, color, and pattern. This means that the clothing product generation system may not be able to achieve the requirement of multi-region content fusion while maintaining the original characteristics of clothing by relying solely on the existing image generation system. Therefore, it is necessary to propose a new method to improve the controllability of the image based on the original image generation system. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and propose a clothing-driven image multi-region feature fusion system based on AI technology.

[0004] First, a system architecture for fusion of multi-region features from fashion trend graphs to clothing product images is provided, including: an improved CLIP graph search model, a SAM segmentation model, and a generative image model based on IP-Adapter-MultiDiffusion;

[0005] Among them, the clothing image, fashion trend map and composition reference map are respectively input into an image encoder to obtain a vector representation of the image; the IP-Adapter-MultiDiffusion model uses the SDXL pre-training model.

[0006] In a second aspect, a clothing-driven image multi-region feature fusion system based on AI technology as described in the first aspect is provided, comprising:

[0007] Step 1, constructing a clothing-driven image multi-region feature fusion model, wherein the clothing-driven image multi-region feature fusion model includes an optimized CLIP image search model, a SAM model and an improved MultiDiffusion model;

[0008] Step 2: Optimize the CLIP image search model to find the clothing parts of the clothing pictures in the clothing style library, and match the clothing parts with the fashion trend map, so as to search for clothing images that are more matched with the fashion trend map.

[0009] Step 3: Improve MultiDiffusion and IP-Adapter technologies to design a multi-region fusion algorithm. That is to say, the MultiDiffusion denoising network structure is adjusted accordingly, and the cropping and background replacement processing layers are added. The adjusted MultiDiffusion model can realize multi-person + environment dressing, that is, multiple clothes are migrated to the specified image area.

[0010] Preferably, in step 1, the CLIP model is used to search for a group of clothing images with high matching degree with the trend graph from the style library according to the trend graph. The trend graph and the style library provided by the user are respectively generated their vector representations through the image encoder of CLIP, and the cosine similarity scores of the trend graph and the images in the style library are calculated. The scores are matched from low to high, and the user can select the first n images that best match the trend graph.

[0011] Preferably, in step 1, the SAM model is used to automatically identify all objects in the image and generate a mask, and the clothing image is cropped to a suitable size according to the generated mask. The processed clothing image will be used as an input image for the IP-Adapter technology.

[0012] Preferably, in step 1, the Multidiffusion Unet model consists of three parts. The input of the Unet model includes three parts: an image obtained by splicing the clothing image and the reference composition image, a mask obtained by splicing the clothing image and the reference composition image mask, and an image encoder to a latent space. The basic model of the Multidiffusion model is the SDXL pre-trained model, which is a pre-trained model:

[0013]

[0014] Its image space is I = R H×W×C , the conditional space is The denoising sampling process is:

[0015] I T ,I T-1 ,…,I0 stI t-1 =Φ(I t |y)

[0016] This process gradually transforms the Gaussian noise I T becomes a clean image I0. Let the region mapping between each target image space and the base model image space be F:J→I, where J is the target image space. If F(J t ) is as close as possible to the sampling Φ(I t|y), the overall image can be seamlessly integrated while keeping the regional content in compliance with the image prompts.

[0017] Preferably, in step 1, the IP-Adapter uses the CLIP image encoder to extract feature information of the selected image with only the clothing part left after cropping, and fuses it with the cross attention layer to enhance the understanding of clothing features.

[0018] The IP-Adapter formula is as follows:

[0019] Attention(Q,K c ,V c )+Attention(Q,K i ,V i ),

[0020] in, Q∈R n×d is the query matrix from MultidiffusionUnet, K c ∈R n×d , V c ∈R n×d are the key-value pairs from the text embedding c, K i ∈R n×d , V i ∈R n×d are the key-value pairs from the clothing image embedding i, respectively.

[0021] Preferably, step 2 comprises:

[0022] Step 2.1.1, apply the SAM style model to segment clothing parts from clothing style library images;

[0023] Step 2.1.2: Input the segmented images into an image encoder to obtain the vector representation of the image I f , input the color category text and the popular trend picture into a text encoder and an image encoder respectively, and obtain the vector representation RT of the color category text and the popular trend picture f and RI f ;

[0024] Step 2.1.3: Map the image and text vector representations into a joint multimodal space to obtain new image and text vector representations I that can be directly compared. e , RT e and RI e, calculate the Cosine similarity between clothing images and color category text, fashion trend pictures and color category text, clothing images and fashion trend picture vectors. Calculate the Cosine similarity between fashion trend pictures and color category text, get the color closest to the color category text in the fashion trend picture, and then use the obtained color as the color category text of the clothing picture and the clothing picture vector to calculate the Cosine similarity between the vectors. The similarity between clothing pictures and fashion trend pictures is expressed as:

[0025] Cos=aCos i→t +bCos i→i

[0026] Among them, Cos represents the Cosine similarity between clothing and fashion trend pictures, Cos i→t Indicates the Cosine similarity between clothing images and color category texts, Cos t→t It represents the Cosine similarity between clothing pictures and fashion trend pictures. a<1 and b<1-a are the weights from image to text and image to image, respectively. The objective function of contrastive learning is to make the similarity of positive sample pairs higher and the similarity of negative samples lower.

[0027] Preferably, in step 3, the optimization problem is expressed as:

[0028]

[0029] Among them, M i is a set of masks generated by the SAM model, y i is a set of corresponding image prompts, J t It is the tth sampling process of Multidiffusion.

[0030] Step 3.1.1: The above optimization problem is also a least squares problem, and its optimal solution can be calculated by analytical method. i is the process of directly cropping the masked area from the target image. In this case, each pixel of the optimal solution of J is the weighted average of all diffusion samples of the base model:

[0031]

[0032] Step 3.1.2: Considering that the structural layout is determined early in the diffusion process, Φ(J t |y i ) Focus on region M early in the generation process i To match the desired layout, the overall coherence of the background and the generated clothing is then considered. Based on this, the region map is defined as:

[0033]

[0034] Among them, I background is the background image without all clothing masks. init It can be set to 20% of the generation process. In the first 20% of the generation process, the clothing pictures and modified pictures are in the solid color background of the original environment of the modified pictures, and the last 80% are replaced back to the background image of the composition reference picture.

[0035] Step 3.1.3: In order to apply the Multidiffsion model described in step 3, the image needs to be preprocessed. The output image and the clothing image should be in the same background image. If the clothing image is not in the same environment as the output image, the background of the clothing image should be replaced with a solid color background similar to the background of the output image, as described in step 3.1.2.

[0036] Step 3.1.4, as described in step 3.1.3 above, this application adds a cropping and splicing layer in the MultiDiffusion model, and adds a cropping and splicing layer in the Self-Attention layer to crop the image area to be processed and extract the reference clothing image as the input of the IP-Adapter.

[0037] The beneficial effects of the present invention are:

[0038] The present invention provides an improved image generation model, which adds cropping and splicing layers in the denoising network, while ensuring that multiple garments are migrated to the specified area of ​​the picture, and realizing the overall fusion of the entire image.

[0039] The present invention provides an end-to-end multi-region image generation model from trend graph to product graph, which accelerates the user design of clothing product graph and feedback process. The user only needs to provide the trend graph, clothing style library, structure reference graph and the preset number of clothing pictures. The model can automatically select a specified number of clothing that highly matches the trend graph from the clothing style library according to the provided information, and then seamlessly sew the selected clothing into the specified area of ​​the reference structure graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall system structure;

[0041] Figure 2 This is a schematic diagram of the image search process;

[0042] Figure 3 Generate model diagram for MultiDiffusion;

[0043] Figure 4 These are the clothing pictures and target pictures of Example 1. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0045] The algorithm flow is as follows Figure 1 shown.

[0046] Embodiment 1:

[0047] The embodiment of the present application first analyzes the needs of the process from trend chart to clothing product map. The fashion trend chart can be a fashion trend chart provided by a fashion trend forecasting website in the public network, and the clothing product map mainly includes a single-person single clothing product map, a multi-person multi-clothing product map, etc. Taking the 2025 color trend and multi-person multi-clothing product map provided by the WGSN fashion trend forecasting website as an example, a fashion trend chart, a clothing style library, and a composition reference map are required.

[0048] Step 1: Input the fashion trend chart and style library, and search for the clothing that best matches the fashion trend chart from the style library, such as Figure 2 As shown, select a clothing picture;

[0049] Step 2: Input the composition reference image and clothing image, and use the generative model to generate the target image, such as Figure 4 shown.

Claims

1. A clothing-driven image multi-region feature fusion system based on AI technology, characterized in that: include: Improved CLIP image search model, SAM segmentation model, and clothing image-driven image multi-region generation model based on IP-Adapter-MultiDiffusion; The clothing image, fashion trend map and composition reference map are respectively input into an image encoder to obtain a vector representation of the image; the pre-trained model used by the IP-Adapter-MultiDiffusion model is the SDXL image model.

2. A clothing-driven image multi-region feature fusion system based on AI technology as claimed in claim 1, characterized in that: include: Step 1, constructing a clothing-driven image multi-region feature fusion model, wherein the clothing-driven image multi-region feature fusion model includes an optimized CLIP image search model, a SAM model and an improved MultiDiffusion model; Step 2: Optimize the CLIP image search model to find clothing parts of clothing images in the clothing style library and match the clothing parts with the fashion trend map, so as to search for clothing images that are more matched with the fashion trend map as recommended clothing; Step 3: Improve MultiDiffusion and IP-Adapter technologies to design a multi-region fusion algorithm. The MultiDiffusion denoising network structure was adjusted accordingly, and a new processing layer was added to replace the target image background, crop the target image mask area and splice the cropped target image with the clothing image as the target image, and splice the clothing image as the input image of the IP-Adapter encoder and the processing mask image. The adjusted MultiDiffusion model can realize multi-person + environment dressing, and can realize the simultaneous migration of multiple clothing to the specified image area.

3. The AI-based clothing-driven image multi-region feature fusion system according to claim 2 is characterized in that: In step 1, the CLIP model acts as follows: It is used to search for a set of clothing images with high matching degree with the trend graph from the style library according to the trend graph. The trend graph and style library provided by the user are respectively generated by the image encoder of CLIP to generate their vector representations, and the cosine similarity scores of the trend graph and the images in the style library are calculated. The scores are matched from low to high, and the user can select the first n images that best match the trend graph.

4. The AI-based clothing-driven image multi-region feature fusion system according to claim 3 is characterized in that: In step 1, the SAM model: It is used to automatically identify all clothing objects in the image and generate masks, and then crop the clothing image to a suitable size according to the generated masks. The processed clothing image is used as the input image of the clothing image of the image search model and the input image of the IP-Adapter encoding layer of the IP-Adapter-MultiDiffusion model.

5. The AI-based clothing-driven image multi-region feature fusion system according to claim 4 is characterized in that: In step 1, the Multidiffusion Unet model consists of three parts: The input of the Unet model consists of three parts: the image after the clothing image and the reference composition image are spliced, the mask after the clothing image and the reference composition image mask are spliced, and then the image encoder is used to convert the mask into a latent space. The basic model of the Multidiffusion model is the SDXL pre-trained model, which is a pre-trained model: Φ:I×Υ→I Its image space is I = R H×W×C , the conditional space is Υ, and the denoising sampling process is: AND T ,AND T-1 ,...,I0s.tI t-1 =Φ(I t |y) This process gradually transforms the Gaussian noise I T becomes a clean image I0. Let the region mapping between each target image space and the base model image space be F:J→I, where J is the target image space. If F(J t ) is as close as possible to the sampling Φ(I t |y), the overall image can be seamlessly integrated while keeping the regional content in compliance with the image prompts.

6. The AI-based clothing-driven image multi-region feature fusion system according to claim 5 is characterized in that: In step 1, the IP-Adapter: The CLIP image encoder is used to extract the feature information of the selected image with only the clothing part left after cropping, and it is fused with the cross attention layer to enhance the understanding of clothing features. The IP-Adapter formula is as follows: Attention(Q,K c ,V c )+Attention(Q,K i ,V i ), in, Q∈R n×d is the query matrix from Multidiffusion Unet, K c ∈R n×d , V c ∈R n×d are the key-value pairs from the text embedding c, K i ∈R n×d , V i ∈R n×d are the key-value pairs from the clothing image embedding i, respectively.

7. The AI-based clothing-driven image multi-region feature fusion system according to claim 6, characterized in that: In step 3, the optimization problem is expressed as: Among them, M i is a set of masks generated by the SAM model, y i is a set of corresponding image prompts, which are image embeddings of clothing images through IP-Adapter adapters, J t This is the tth sampling process of Multidiffusion. With the addition of IP-Adapter, the Multidiffusion model that only supports generating images from text can now generate images from images.

8. The AI-based clothing-driven image multi-region feature fusion system according to claim 7, characterized in that: Step 2 includes: Step 2.1.1, apply the SAM style model to segment clothing parts from clothing style library images; Step 2.1.2: Input the segmented images into an image encoder to obtain the vector representation of the image I f , input the color category text and the popular trend picture into a text encoder and an image encoder respectively, and obtain the vector representation RT of the color category text and the popular trend picture f and RI f ; Step 2.1.3: Map the image and text vector representations into a joint multimodal space to obtain new image and text vector representations I that can be directly compared. e , RT e and RI e , calculate the Cosine similarity between clothing images and color category text, fashion trend pictures and color category text, clothing images and fashion trend picture vectors. Calculate the Cosine similarity between fashion trend pictures and color category text, get the color closest to the color category text in the fashion trend picture, and then use the obtained color as the color category text of the clothing picture and the clothing picture vector to calculate the Cosine similarity between the vectors. The similarity between clothing pictures and fashion trend pictures is expressed as: Cos=aCos i→t +bCos i→i Among them, Cos represents the Cosine similarity between clothing and fashion trend pictures, Cos i→t Indicates the Cosine similarity between clothing images and color category texts, Cos t→t It represents the Cosine similarity between clothing pictures and fashion trend pictures, a<1 and b<1-a are the weights from image to text and image to image respectively. The objective function of contrastive learning is to make the similarity of positive sample pairs higher and the similarity of negative samples lower.

9. The AI-based clothing-driven image multi-region feature fusion system according to claim 8, characterized in that: Step 3 includes: Step 3.1.1: The above optimization problem is also a least squares problem, and its optimal solution can be calculated by analytical method. i is the process of directly cropping the masked area from the target image. In this case, each pixel of the optimal solution of J is the weighted average of all diffusion samples of the base model: Step 3.1.2: Considering that the structural layout is determined early in the diffusion process, Φ(J t |y i ) Focus on region M early in the generation process i To match the desired layout, the overall coherence of the background and the generated clothing is then considered. Based on this, the region map is defined as: Among them, I background is the background image without all clothing masks. init It can be set to 20% of the generation process. In the first 20% of the generation process, the clothing pictures and modified pictures are in the solid color background of the original environment of the modified pictures, and the last 80% are replaced back to the background image of the composition reference picture. Step 3.1.3: In order to apply the Multidiffsion model described in step 3, the image needs to be preprocessed. The output image and the clothing image should be in the same background image. If the clothing image is not in the same environment as the output image, the background of the clothing image should be replaced with a solid color background similar to the background of the output image, as described in step 3.1.

2. Step 3.1.4, as described in step 3.1.3 above, this application adds a cropping and splicing layer in the MultiDiffusion model, and adds a cropping and splicing layer in the Self-Attention layer to crop the image area to be processed and extract the reference clothing image as the input of the IP-Adapter.

10. A clothing-driven image multi-region feature fusion system based on AI technology, characterized in that: The system for fusion of multi-region features of clothing-driven images based on AI technology is used to execute any one of claims 2 to 9, comprising: Construction module: used to construct a clothing-driven image multi-region feature fusion system based on AI technology, wherein the clothing-driven image multi-region feature fusion system based on AI technology includes a CLIP image search model, a SAM segmentation model, and an image multi-region generation model driven by clothing images based on IP-Adapter-MultiDiffusion. Establishing module: used to establish an image generation model with a denoising network with splicing and cropping modules according to the clothing image driven image multi-region generation model based on IP-Adapter-MultiDiffusion.

11. A denoising network with a splicing and cropping module, characterized in that: The splicing and cropping module adds a MultiDiffusion denoising network to replace the image background to avoid the influence of other areas on the target area and enhance the model's understanding of the detailed features of clothing to retain the detailed effects of clothing; the target image spliced ​​with the clothing image and the reference image spliced ​​with the clothing image obtained by the splicing and cropping module are respectively input into the self-attention layer and cross-attention layer of the MultiDiffusion denoising network, so that the model executes the AI ​​technology-based clothing-driven image multi-region feature fusion system described in any one of claims 2 to 8.

Images