An AI technology-based clothing-driven image multi-region feature fusion system

Abstract

The application provides a clothing-driven image multi-region feature fusion system based on AI technology, and relates to the technical field of image generation. The system combines AI technology, first applies a SAM image segmentation model to extract the clothing part of the image. Next, a CLIP model is used to search for clothing images with high similarity to the trend graph from the style library. Then, an improved Multidiffusion image generation model is used to generate a high-quality and high-fusion clothing product image. In the Unet denoising network of the Multidiffusion image generation model, an image cutting and splicing processing layer is added to enhance the understanding of the model to the clothing image features and eliminate the influence of the background part on the generated region in the early stage of the sampling process. Finally, the background image is replaced with the original background image in the late stage of the image sampling process to generate a high-precision and coherent clothing product image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image generation, more specifically, it relates to a clothing-driven image multi-region feature fusion system based on AI technology. BACKGROUND

[0002] Although the Stable Diffusion model provides more generalized content generation capabilities for the development of artificial intelligence technology to automatically generate content, there are still some challenges in practical applications, such as the need for clothing product images to be automatically generated in user-specified image regions without changing clothing structure, color, and pattern features. This means that the clothing product generation system may not be able to meet the demand for multi-region content fusion based on maintaining the original features of the clothing by relying solely on existing image generation systems, so new methods are needed to improve the controllability of the picture based on the original image generation system. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art and provide a clothing-driven image multi-region feature fusion system based on AI technology.

[0004] In a first aspect, a system architecture for generating clothing product images from popular trend images is provided, including an improved CLIP image search model, a SAM segmentation model, and a generated image model based on IP-Adapter-MultiDiffusion.

[0005] Wherein the clothing image, the popular trend image and the composition reference image are input into an image encoder to obtain the vector representation of the image; the IP-Adapter-MultiDiffusion model uses an SDXL pre-training model.

[0006] In a second aspect, a clothing-driven image multi-region feature fusion system based on AI technology is provided, including:

[0007] Step 1, constructing a clothing-driven image multi-region feature fusion model, the clothing-driven image multi-region feature fusion model includes an optimized CLIP image search model, a SAM model and an improved MultiDiffusion model;

[0008] Step 2, the optimized CLIP image search model searches for clothing images that match the popular trend image by matching the clothing part of the clothing image in the clothing style library with the popular trend image.

[0009] Step 3, an improved multi-region fusion algorithm is designed by improving the two technologies of MultiDiffusion and IP-Adapter. That is, the MultiDiffusion denoising network structure is adjusted accordingly, and the processing layers of cropping and background changing are added. After adjustment, the MultiDiffusion model can realize multi-person + environment dressing, that is, multiple clothes are migrated to the specified image region.

[0010] As preferred, in step 1, the CLIP model is used to search a set of clothes images with high matching degree from the style library according to the trend graph. The trend graph and the style library provided by the user are respectively generated through the image encoder of CLIP to obtain their vector representations, the cosine similarity scores of the images in the trend graph and the style library are calculated, the scores are matched from low to high, and the user can select the top n images most matched with the trend graph.

[0011] As preferred, in step 1, the SAM model is used to automatically identify all objects existing in the image and generate a mask, and the clothes image is cropped to a proper size according to the generated mask, and the processed clothes image will be used as the input image of the IP-Adapter technology.

[0012] As preferred, in step 1, the Multidiffusion Unet model is composed of 3 parts. The input of the Unet model includes 3 parts: the image obtained by splicing the clothes image and the reference composition image, the mask obtained by splicing the clothes image and the reference composition image mask, and the image encoder to the latent space. The basic model of the Multidiffusion model is the SDXL pre-training model, which is a pre-training model:

[0013] Φ: I x Y→I

[0014] Its image space is I = R H×W×C , and the condition space is Y. The denoising sampling process is:

[0015] I T ,I T-1 ,…,I0s.t.I t-1 =Φ(I t |y)

[0016] This process gradually changes the Gaussian noise I T into a clean image I0. Let the region mapping between each target image space and the basic model image space be F: J→I, and J be the target image space. If F(J t ) is as close as possible to the sampling Φ(I t |y) of the basic model, the region content is kept in compliance with the image prompt, and the overall image is seamlessly fused.

[0017] As preferred, in step 1, the IP-Adapter adapter extracts the feature information of the selected image of the remaining garment part after cropping using the CLIP image encoder and fuses it with the cross-attention layer to enhance the understanding of the garment features.

[0018] The IP-Adapter formula is as follows:

[0019] Attention(Q, K c , V c ) + Attention(Q, K i , V i ),

[0020] wherein, is the query matrix from Multidiffusion Unet, K c ∈R n×d , V c ∈R n×d are the key-value pairs from text embedding c, K i ∈R n×d , V i ∈R n×d are the key-value pairs from garment image embedding i, respectively.

[0021] As preferred, step 2 includes:

[0022] Step 2.1.1, applying a SAM style model to segment the garment part from the garment style library picture;

[0023] Step 2.1.2, inputting the segmented image into an image encoder respectively to obtain the vector representation I f , inputting the color category text and popular trend picture into a text encoder and image encoder respectively to obtain the vector representation RT f and RI f of the color category text and popular trend picture;

[0024] Step 2.1.3, mapping the image and text vector representations to a joint multi-modal space to obtain new image and text vector representations I e , RT e and RI e, the Cosine similarity between the clothing image and the color category text, the popular trend picture and the color category text, and the clothing image and the popular trend picture vector is calculated. The Cosine similarity between the popular trend picture and the color category text is calculated to obtain the color in the popular trend picture that is closest to the color category text, and then the obtained color is taken as the color category text of the clothing picture to calculate the Cosine similarity between the clothing picture vector and the clothing picture vector. The similarity between the clothing picture and the popular trend picture is represented as:

[0025] Cos = aCos i→t + bCos i→i

[0026] wherein Cos represents the Cosine similarity between the clothing and the popular trend picture, Cos i→t represents the Cosine similarity between the clothing picture and the color category text, Cos t→t represents the Cosine similarity between the clothing picture and the popular trend picture, a < 1 and b < 1-a are the weights of the image-to-text and the image-to-image, respectively. The objective function of the contrast learning is to make the similarity of the positive sample pair higher and the similarity of the negative sample pair lower.

[0027] Preferably, in step 3, the optimization problem is represented as:

[0028]

[0029] wherein M i is a set of masks generated by the SAM model, y i is a set of corresponding image prompts, and J t is the t-th sampling process of the Multidiffusion.

[0030] Step 3.1.1, the above optimization problem is also a least squares problem, and its optimal solution can be calculated by an analytical method. In the present application, F i is a process of directly cropping the mask region from the target image, and in this case, each pixel of the optimal solution of J is a weighted average of all diffusion samplings of the base model:

[0031]

[0032] Step 3.1.2, considering that the structural layout is determined early in the diffusion process, Φ(J t | y i ) should be focused on the region M i in the early stage of the generation process to match the required layout, and then the background and the overall coherence of the generated clothing are considered. Based on this, the region mapping is defined as:

[0033]

[0034] wherein, I background is the background image with all clothing masks removed. T init can be set to 20% of the generation process, in the first 20% of the generation process, both the clothing picture and the modified picture are in the solid color background of the original environment of the modified picture, and the last 80% is replaced back to the background image of the composition reference picture.

[0035] Step 3.1.3, in order to apply the Multidiffsion model described in step 3, the picture needs to be preprocessed, and the output picture and the clothing picture should be in the same background picture. If the clothing picture is not in the same environment as the output picture, the background of the clothing picture will be replaced with a solid color background similar to the background of the output picture, as described in step 3.1.2.

[0036] Step 3.1.4, as described in step 3.1.3 above, the present application adds a cropping and splicing layer in the MultiDiffusion model, and adds a cropping and splicing layer in the Self-Attention layer to crop the image area that needs to be processed and extract the reference clothing image as the input of the IP-Adapter.

[0037] The beneficial effects of the present application are:

[0038] In the present application, an improved image generation model is provided, which adds a cropping and splicing layer in the denoising network. While ensuring the migration of multiple clothes to the specified area of the picture, the whole image is integrated.

[0039] The present application provides an end-to-end multi-region image generation model from trend charts to product charts, which accelerates the user's design of clothing product charts and feedback process. The user only needs to provide trend charts, clothing style libraries, structure reference charts and preset clothing picture quantities, and the model can automatically select a specified number of clothes that match the trend chart from the clothing style library according to the provided information, and then seamlessly stitch the selected clothes in the specified area of the reference structure chart. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic diagram of the overall system structure;

[0041] Figure 2 is a schematic diagram of the image search process;

[0042] Figure 3 is a schematic diagram of the MultiDiffusion generation model. DETAILED DESCRIPTION

[0043] The application will be further described below in connection with the embodiments. The following description of the embodiments is only to help understand the application. It should be pointed out that for ordinary people in the art, several modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

[0044] The algorithm flow is as shown in Figure 1 .

[0045] Embodiment 1

[0046] The embodiment of the present application firstly analyzes the demand for the process from the trend chart to the production of the clothing product chart. The popular trend chart can be the popular trend chart provided by the popular trend prediction website in the public network, and the clothing product chart mainly includes single-person single-clothing product chart, multi-person multi-clothing product chart and the like. Taking the color trend in 2025 and the multi-person multi-clothing product chart provided by the WGSN popular trend prediction website as an example, the popular trend chart, the clothing style library and the composition reference chart need to be used.

[0047] Step 1, input the popular trend chart and the style library, search the most matched clothing from the style library, as shown in Figure 2 , select a clothing chart;

[0048] Step 2, input the composition reference chart and the clothing chart, and generate the target picture by using the generation model.

Claims

1. An AI technology-based clothing-driven image multi-region feature fusion system, characterized in that, The improved CLIP image search model, the SAM segmentation model, and the image multi-region generation model driven by a clothing image based on the IP-Adapter-MultiDiffusion are included. The improved CLIP image search model is used to generate vector representations of a popular trend image provided by a user and clothing images in a style library through an image encoder of the improved CLIP image search model respectively, calculate cosine similarity scores of the popular trend image and the clothing images in the style library, and match the scores from low to high for the user to select the top n clothing images that are most matched with the popular trend image. The SAM segmentation model is used to automatically identify all clothing objects in the clothing images and generate masks, and cut the clothing images to a proper size according to the generated masks. The image multi-region generation model is used to receive the clothing images and the masks, and a reference composition image provided by a user. The image multi-region generation model includes a Multidiffusion Unet model and an IP-Adapter adapter. The input of the Multidiffusion Unet model includes three parts: a spliced image of the clothing image and the reference composition image, a spliced mask of the clothing image and the reference composition image mask, and hidden space features obtained by compressing the spliced image and the spliced mask through an image encoder. The spliced image and the spliced mask are denoised and generated in the hidden space, and the target region background is replaced, the mask is cut, and the pixel-level splicing with the clothing image is completed in the processing layer of the Multidiffusion Unet model to form an intermediate synthesis image. The intermediate synthesis image is introduced into a cross-attention layer as a clothing feature reference through the IP-Adapter adapter to drive the Multidiffusion Unet model to fuse clothing details only in the specified mask region, realize multi-person / multi-region synchronous dressing, and output a generated image that is consistent with the reference composition image in size, background, and character posture and has seamlessly replaced the specified clothing. The pre-training model used by the Multidiffusion Unet model is an SDXL image large model. 2.The AI technology-based system for fusing multi-region features of a garment-driven image according to claim 1, wherein, The generation of vector representations of the popular trend image provided by the user and the clothing images in the style library through the image encoder of the improved CLIP image search model and the calculation of the cosine similarity scores of the popular trend image and the clothing images in the style library include: 3.The AI technology-based system for fusing multi-region features of a garment-driven image according to claim 1, wherein, Step 1: applying the SAM segmentation model to segment the clothing part from the clothing images in the style library. ​ Step 2, inputting the segmented garment image into the image encoder to obtain a vector representation I of the segmented garment image f inputting the color category text and the trend graph into a text encoder and the image encoder respectively to obtain a vector representation RT of the color category text and the trend graph f and RI f ; Step 3, map I f , RT f and RI f to a joint multimodal space, resulting in new directly comparable vector representations I e , RT e and RI e , The Cosine similarity between the segmented clothing image and the color category text, the popular trend graph and the color category text, and the segmented clothing image and the popular trend graph vector is calculated, including: calculating the Cosine similarity between the popular trend graph and the color category text vector, obtaining the color closest to the color category text in the popular trend graph, and then taking the obtained color as the Cosine similarity between the color category text of the segmented clothing image and the segmented clothing image; the Cosine similarity between the segmented clothing image and the popular trend graph vector is represented as: Cos = aCos i→t + bCos i→i wherein Cos i→t denotes the Cosine similarity between the segmented garment image and the color category text vector, Cos i→i denotes the Cosine similarity between the segmented garment image and the trend graph vector, a < 1 and b < 1-a are the weights of image-to-text and image-to-image, respectively.

Images