Instruction-driven personalized fashion image editing method

Abstract

The invention discloses an instruction-driven personalized fashion image editing method. The method comprises the following steps: 1, defining categories of editing tasks, and constructing an original image-reference image-target image-text editing instruction quaternary data set for different editing tasks; 2, constructing a target semantic network, generating target image semantic information following an editing instruction and an original image, and taking the target image semantic information as human body semantic information of an editing model; 3, constructing a unified editing network, including constructing a semantic control network, adding a visual joint module and applying a low-rank fine tuning module, so that a plurality of editing tasks use the same framework to obtain corresponding editing capabilities; and 4, constructing a multi-task low-rank adjustment module, and enabling the framework to have the capability of aligning different editing instructions to different editing tasks through joint training. Finally, an independent and unified framework among different tasks is realized. According to the method, experiments are carried out on a constructed specific data set, and good results are obtained in both quantitative and qualitative aspects.

Description

Technical Field

[0001] The invention belongs to the technical field of image editing, and in particular relates to an instruction-driven personalized fashion image editing method.

[0002] The present invention mentions a new method of instruction-based customized fashion image editing. It mainly involves a low-rank fine-tuning technology applied to a pre-trained model to understand and implement the editing intention for different editing tasks; and designs a multi-task low-rank global selection mechanism to provide a unified framework for the implementation of different editing tasks. In this way, multiple editing tasks can be compatible under one framework, and real and effective fashion image editing results can be obtained. Background Art

[0003] In recent years, the rapid development of diffusion models and the evolution of controllable generation technology have made great progress in the authenticity, controllability and diversity of image editing technology. As an important part of image editing technology, fashion editing tasks pay more attention to the preservation of human details and the reconstruction of fashion features. Therefore, they have always received high attention in industry and academia.

[0004] Fashion image editing tasks have a certain difference in focus from traditional image editing tasks. The specific editing tasks solved by previous work mainly include two-dimensional posture migration, fashion attribute modification, and human-centered clothing reconstruction. The driving modes are mainly text-driven, image-driven, and multimodal-driven. In order to cover as many editing tasks as possible and have the most compatible driving mode, this invention attempts to design a unified fashion image editing framework with three characteristics: instruction-driven, personalized support, and task compatibility. However, the constraints of real resources and the limitations of technical conditions pose many challenges.

[0005] 1) Lack of fashion editorial datasets In order to overcome the limitations of the original generation framework, such as unintended editing, fashion editing tasks require original images, editing instructions, and high-quality edited images that follow the editing instructions as training data. Complete editing image and text data pairs are difficult to collect in Internet data due to their sparseness. The existing original data is difficult to use directly in terms of integrity and quality, and there is a large skew in categories, which is not conducive to model training. Therefore, it is key to construct a large number of high-quality, wide-category, and weakly skewed datasets for various editing tasks.

[0006] Framework design compatible with different editing tasks Different editing tasks may require different auxiliary information to enhance editing controllability. Therefore, it is necessary to determine a combination of auxiliary information compatible with different tasks to achieve editing guidance between different tasks.

[0007] Effective Fusion Training Strategy There are obvious differences in editing intentions between different editing tasks. Experiments have shown that simple sample mixing training cannot enable the generative model to achieve different editing tasks while maintaining prior knowledge. Specifically, the editing results show the effects of other editing tasks, which is called intention confusion. Therefore, effectively overcoming intention confusion and enabling the model to be compatible with different editing intentions has become an important challenge for the unified fashion editing framework.

[0008] The present invention is committed to integrating multiple editing capabilities into a unified framework through a small amount of training on the basis of the existing pre-trained image generation model, so as to generate fashion image editing results with high degree of freedom, high authenticity and high accuracy. Summary of the invention

[0009] The present invention provides a command-driven personalized fashion image editing method, which constructs a multi-editing task dataset, designs a unified framework for multi-editing tasks, trains multiple groups of task-independent editing modules, and unifies the multi-editing tasks through a multi-task low-rank global selection mechanism, thereby generating realistic and accurately edited fashion images. The invention has been experimented on a specific dataset and has achieved good results both quantitatively and qualitatively.

[0010] A command-driven personalized fashion image editing method, the steps of which are as follows: Step (1), defining the categories of editing tasks, and constructing a four-dimensional data set of "original image - reference image - target image - text editing instruction" by using web crawler technology and existing image generation technology for different editing tasks; Step (2), constructing and training a target semantic network; using the text editing instructions, the reference image and the original image as inputs of the target semantic network, extracting the semantic information of the target image as a training target, obtaining the semantic information of the target image following the text editing instructions, and using the semantic information of the target image as the human body semantic information of the editing model; Step (3), build a unified editing network; add three changes based on the pre-trained instruction-driven generation model: 1. Build a semantic control network to achieve effective guidance based on the semantic information of the target image obtained in step (2); 2. Add a visual joint module to reconstruct the reference image in the editing result; 3. Apply a low-rank fine-tuning module to align the understanding and implementation of editing intent to the low-rank fine-tuning module through a small amount of training; thereby effectively achieving multiple editing tasks using the same framework to obtain corresponding editing capabilities; Step (4): construct a multi-task low-rank fine-tuning module; through joint training, enable the framework to have the ability to align different text editing instructions to different editing tasks; finally, realize an independent yet unified fashion editing framework between different tasks.

[0011] Beneficial effects of the present invention: The present invention proposes a command-driven personalized fashion image editing method. By constructing a multi-editing task dataset, designing a unified editing framework and a low-rank global selection mechanism, the integration of multiple editing tasks and high-quality editing effects are achieved. Through the multi-task low-rank global selection module, the independence and unity between different editing tasks are achieved, and the problem of decentralized and difficult integration of multi-task processing in existing methods is solved. By using the semantic control network and the visual joint module, the editing implementation capability is significantly improved, and the generated fashion images have realistic effects and high personalization. Through low-rank fine-tuning technology and a unified framework, a variety of tasks can be quickly adapted through a small amount of training, making the method efficient, flexible and scalable. Experiments show that the method achieves excellent performance in both quantitative and qualitative evaluations, and is suitable for personalized clothing matching, virtual try-on, image editing and other scenarios in e-commerce and fashion industries, and has a wide range of practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a specific schematic diagram of the process of the present invention.

[0013] Figure 2 It is a schematic diagram of the target semantic network in the present invention.

[0014] Figure 3 It is a schematic diagram of the low-rank fine-tuning module in the method of the present invention.

[0015] Figure 4 It is a schematic diagram of the multi-task low-rank fine-tuning module in the method of the present invention.

[0016] Figure 5 It is a schematic diagram of the unified editing network of the present invention.

[0017] Figure 6 It is a schematic diagram of the data set of the present invention.

[0018] Figure 7 It is a schematic diagram of the effect of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] The present invention provides a command-driven personalized fashion image editing method, such as Figure 1As shown, the method constructs a multi-editing task dataset, designs a unified framework for multi-editing tasks, trains multiple sets of task-independent editing modules, and unifies multiple editing tasks through a multi-task low-rank global selection mechanism to generate realistic and accurately edited fashion images. The invention was experimented on a specific dataset and achieved good results both quantitatively and qualitatively.

[0021] A command-driven personalized fashion image editing method, the steps of which are as follows: Step (1), defining the categories of editing tasks, and constructing a four-dimensional data set of "original image - reference image - target image - text editing instruction" by using web crawler technology and existing image generation technology for different editing tasks; Step (2), constructing and training a target semantic network; using the text editing instructions, the reference image and the original image as inputs of the target semantic network, extracting the semantic information of the target image as a training target, obtaining the semantic information of the target image following the text editing instructions, and using the semantic information of the target image as the human body semantic information of the editing model; Step (3), build a unified editing network; add three changes based on the pre-trained instruction-driven generation model: 1. Build a semantic control network to achieve effective guidance based on the semantic information of the target image obtained in step (2); 2. Add a visual joint module to reconstruct the reference image in the editing result; 3. Apply a low-rank fine-tuning module to align the understanding and implementation of editing intent to the low-rank fine-tuning module through a small amount of training; thereby effectively achieving multiple editing tasks using the same framework to obtain corresponding editing capabilities; Step (4): construct a multi-task low-rank fine-tuning module; through joint training, enable the framework to have the ability to align different text editing instructions to different editing tasks; finally, realize an independent yet unified fashion editing framework between different tasks.

[0022] The categories of the defined editing tasks described in step (1) are add, delete, replace, and modify. Through web crawler technology and various existing advanced image synthesis technologies, a large number of noisy image and text data pairs were obtained; after manual screening, 36,312 pairs of editing image and text data pairs were finally obtained, of which 10,244 were replacement tasks, 10,072 were modification tasks, and 7,998 were add and delete tasks respectively; from the category point of view, ADCR involves four basic editing tasks, including 8 categories of items: tops, bottoms, full clothes, shoes, glasses, belts, bags, and scarves. The data pairs of different tasks are referenced Figure 6 The final dataset consists of the original images , reference image , text editing commands , target image The target image is composed of a four-tuple; the original image and the reference image are both obtained by the crawler, and the text editing instructions are generated by the large language model based on the comparison between the original image and the target image; the acquisition of the target image is concentrated in the following situations: for adding or deleting tasks, the original image containing the reference image is filtered, and the reference image is removed through SD1.5-Inpaint; the target image in the replacement task uses the existing virtual fitting model; the target image in the modification task is modified by the editing model and then synthesized using the virtual fitting model.

[0023] The image synthesis technology mentioned in step (1) is related to specific editing tasks. For replacing data pairs, objects of the same category as the replacement target are selected as reference images based on the original image, and the target image is obtained through an advanced virtual fitting tool; adding and deleting data pairs are essentially the same, and an advanced image-generated image patching model is used to remove the reference objects in the original image to obtain the target image; limited by the accuracy of existing editing tools, the reference image is modified through the instruction editing model to change its attributes, and then the target image is obtained through the virtual fitting tool, thereby indirectly realizing the modification of the data pair. The original image and the target image are analyzed using an existing multimodal large language model to obtain editing instruction data that is both diverse and accurate. The final dataset consists of the original image , reference image , text editing commands , target image For specific examples, please refer to Figure 6 , where the replacement task and the addition task contain new concepts, so there is a reference image in the data pair; while the deletion task and the modification task do not include a reference image.

[0024] The human semantic information described in step (2) includes several human region labels, and the number of human semantic labels is 18, including background, hat, hair, sunglasses, top, skirt, pants, dress, belt, left shoe, right shoe, face, left leg, right leg, left arm, right arm, bag, and scarf regions, which effectively represent the structural information of the human body and related clothing. The target semantic network is essentially a graph-generated graph model, which uses the stable diffusion model as the basic model and uses the original image in the data pair as the basic model input after forward noise addition. The target semantic network serves as the human semantic guidance prior of the editing model and needs to generate images that follow the text editing instructions. And the target semantic image conforms to the original human semantics , the overall network structure reference Figure 2 .

[0025] The image feature extractor CLIP is used to obtain the 24-layer feature vector of the reference image, i.e., the image embedding. The first 6 layers of feature vectors representing the low-frequency information of the reference image are averaged and then passed through a set of The constructed mapping network is converted to the text embedding space and then mixed with the text embedding to obtain the final text editing instructions containing image features. , thus completing the hybrid embedding.

[0026] The training loss adds a regularization constraint on the feature vector based on the diffusion loss. The specific formula is as follows:

[0027] in, is the VAE variational autoencoder in the stable diffusion model, Represents the latent variable obtained by encoding the input image x through the variational autoencoder , represents the noise distribution extracted from image x, represents the time step, is noise randomly sampled from a normal distribution, Is a function that converts the text template Embedded with images Combined, generate hybrid embeddings for guiding text generation; here represents the final text editing instruction, and is the image embedding, which represents the features extracted from the image; is a hyperparameter used to adjust the strength of the eigenvector regularization constraint; Represents the use of image feature extractor CLIP from the reference image The nth layer feature vector extracted from is a reference image used to provide visual information for training in conjunction with text, Represents a multi-layer perceptron.

[0028] like Figure 5 As shown, the unified editing network described in step (3) is specifically implemented as follows: The original image After being input into the unified editing network, the variational autoencoder that comes with the unified editing network Convert to original latent representation , and then randomly sample noise in the normal distribution as , the initial noise With the original latent representation Concatenate in the channel dimension to obtain potential representation .

[0029] The original image passes through the target semantic network to obtain the target semantic image , the semantic control network transforms the target semantic image After being converted into semantic latent representation, a sequence of semantic feature vectors is extracted , and serves as an input to the unified editing network; After the reference image features are extracted by the image feature extractor CLIP, they are input into the visual joint module of the unified editing network; Latent representation of reference image features in the cross-attention layer of the unified editing network and the editing process To interact with each other, the low-rank fine-tuning module is applied to each attention layer in the unified editing network for each different editing task. 'Step denoising and decoding of the variational autoencoder, the potential representation In the unified editing network The potential representation obtained after denoising , and decode the output as the edited image .

[0030] Part I: Semantic Control Network; The semantic control network uses the encoder part of the generative network as the basic structure, and the initial weights are copies of the corresponding generative network weights; the semantic control network uses the target semantic image As input, take the final text editing command As a text guide; the entire semantic control network is regarded as a semantic information feature extractor, and each time step Each layer Potential manifestations of As the target semantic feature, after being mapped by the corresponding zero convolution layer of each layer, it is used as the corresponding potential representation in the decoder of the residual and generative network Addition guides the generation of target images that conform to text instructions.

[0031] In order to reduce the training burden, the semantic control network will be trained independently from the unified editing network. A self-supervised training method will be adopted for the semantic control network. Specifically, a stable diffusion model is used as the generation network to transform the target image Through variational autoencoder Transform into target potential representation And add noise as the input of the generating network The textual guidance of the denoising process is the target image description generated by a multimodal large language model. ; The semantic control network transforms the target semantic image After being converted into semantic latent representation, a sequence of semantic feature vectors is extracted , and at zero convolution After mapping, the residual is added to the corresponding potential representation layer by layer in the decoder of the generative network to provide semantic guidance for the editing process; the training process freezes all parameters except the semantic control network and only updates all parameters of the semantic control network; the training loss adopts diffusion loss, and the formula is defined as follows:

[0032] in, Represents the potential representation of the input image x after being transformed by the variational encoder (VAE) ; Represents the description of the target image; is the noise sampled from a standard normal distribution, added to the latent representation Afterwards, it is used to generate the denoising process of the image; t represents the time step in the diffusion process; Represents the target semantic image.

[0033] Part II: Visual Joint Module; The visual joint module is an augmented network of the basic cross-attention layer; first, the cross-attention layer in the unified editing network is defined as follows:

[0034] in, Represent the original query features, key features, and value features respectively; refer to Derived from the design of text embedding. The visual joint module proposes a learnable weight matrix and , the reference image extracted by the visual encoder CLIP Embedding maps to visual key features and visual value features .

[0035] Therefore, the hybrid cross attention layer augmented by the visual joint module is defined as follows: .

[0036] in, Represents the dimensions of key and query features. In the attention mechanism, Used to scale the dot product results to prevent gradient instability due to large dimensions. and Represents the key and value features introduced by the visual joint module, which are composed of the reference image Visual features and weight matrices extracted by visual encoders (such as CLIP) and Multiply them together to get .

[0037] Part III: Low-rank fine-tuning module; The low-rank fine-tuning module is applied to separate the editing capabilities of different editing tasks from the basic generation network and transfer them to the corresponding low-rank adapter, so as to achieve the effect that different editing capabilities are independent of each other and do not interfere with each other, as follows: The low-rank fine-tuning module is applied to the mapping layer, self-attention layer, and cross-attention layer in each attention layer of the unified editing network; the low-rank fine-tuning module achieves low-cost fine-tuning by adding low-rank residual networks to all linear layers and convolutional layers, such as Figure 3 As shown, its definition can be abstracted into the following formula:

[0038] in, represents the forward computation of any linear layer or convolutional layer, is the input image of the unified editing network; Represents the weight of the low-rank fine-tuning module, which represents the activation strength of the low-rank fine-tuning module; Represents the weight of the original linear layer or convolutional layer, and its dimension is ; represents the low-rank fine-tuning module, constitute, and The corresponding dimensions are and Linear or convolutional layers, is the rank of the low-rank fine-tuning module.

[0039] Fine-tune the corresponding unified editing network for different editing tasks to obtain a set of low-rank fine-tuning modules corresponding to the editing tasks; freeze all parameters except the low-rank residual network during training, that is, only keep the parameter updates of the low-rank residual network; training loss Including diffusion loss and editing positioning loss, the formula is defined as follows:

[0040]

[0041]

[0042] Formula (8) represents Diffusion loss, where represents the unified editing network after optimization by the low-rank fine-tuning module; Represents the input image After the variational autoencoder The transformed potential representation z; A text description representing the target image; represents the noise sampled from the standard normal distribution; t represents the time step in the diffusion, which controls the progress of the generation process; represents the noisy latent representation at time step t, which serves as the input of the generative network; A latent representation representing the original image; represents the reference image, which is used as a condition for the denoising process; Represents the target semantic image.

[0043] The edit localization loss expressed in formula (9) ; Each layer of cross attention map in the single-step denoising process is defined as , that is, Layer The attention map corresponding to the word unit, the total number of word units is ; Edit the localization loss It is Layer The average attention map of the word unit; because the sizes of attention maps of different layers are not all the same, according to experience, the average attention map The size is uniformly scaled to 16*16 resolution; edit the positioning loss Represents the edit area mask with a resolution of 16*16. The mask data is an augmentation of the basic dataset and is obtained through the SAM segmentation model. Through the edit localization loss, this method focuses the attention of the unified edit network denoising process on the editing subject that follows the text editing instructions, effectively reducing the phenomenon of unintentional editing.

[0044] like Figure 4 As shown in the figure, the multi-task low-rank fine-tuning module in step (4) is essentially an MLP multi-layer perceptron for classification tasks, using text editing instructions The corresponding text embedding is used as input, the feature dimension of the text embedding is aligned with the dimension of the number of tasks, and the output of the multi-task low-rank fine-tuning module is called the low-rank weight ; Weightedly merge the low-rank fine-tuning module sets obtained from independent editing task training, and map the low-rank weights As the activation degree of the low-rank fine-tuning module of the corresponding task; therefore, the abstract definition of the low-rank fine-tuning module in formula (6) is updated to the following formula:

[0045] represents the output generated by the unified editing network; represents the initial linear transformation matrix or weight matrix, which is applied to the input image ; This matrix is ​​fixed during network training and does not participate in low-rank fine-tuning; represents the low-rank weight of the i-th low-rank fine-tuning module, which is the result of the output of the low-rank fine-tuning module SEL; the low-rank weight reflects the activation degree of the low-rank residual network corresponding to the task in each independent editing task; represents the weight change of the i-th low-rank fine-tuning module, that is, the deviation part generated by the low-rank fine-tuning process; , represents the matrix decomposition part of the i-th low-rank fine-tuning module; Among them, the low-rank weight , Represents 4 groups of independently trained low-rank fine-tuning modules; Finally, a joint training strategy is designed to perform end-to-end self-supervised training on the multi-task low-rank fine-tuning module; the training samples used in the joint training are randomly sampled from the "original image-reference image-target image-text editing instruction" four-dimensional data set; in order to make the activation degree of the low-rank fine-tuning module fully consider the balance between the specific editing intensity and the prior knowledge of the editing model, the invention maintains the diffusion loss on the basis of the classification loss, and the specific selection loss The definition is as follows:

[0046] in, Similar to formula (8), but the merged low-rank fine-tuning module calculates the update as formula (10); represents the same text feature extraction module as the unified editing network, namely CLIP; It is a one-hot encoding related to the editing task of the training sample, which ensures that the multi-task low-rank fine-tuning module activates the correct low-rank fine-tuning module while minimizing the activation of the wrong editing ability; represents the low-rank fine-tuning module;

[0047] represents the diffusion loss, where represents the unified editing network after optimization by the low-rank fine-tuning module; Represents the input image After the variational autoencoder The transformed potential representation z; A text description representing the target image; represents the noise sampled from the standard normal distribution; t represents the time step in the diffusion, which controls the progress of the generation process; represents the noisy latent representation at time step t, which serves as the input of the generative network; A latent representation representing the original image; represents the reference image, which is used as a condition for the denoising process; Represents the target semantic image.

[0048] It is worth noting that the unified framework of the invention is highly scalable. When faced with richer fashion editing tasks, such as pose transfer tasks involving global editing, step (3) can also be used to train a low-rank fine-tuning network corresponding to the new task, and the compatibility of the new task can be achieved after simply training the multi-task low-rank global selection module in step (4).

[0049] Example: In actual experiments, this unified editing framework can be used to edit images more accurately. Figure 7 As shown in the figure. The first column is the original image, the second column is the reference image, the third column is the editing instruction, the fourth column is the target semantic image generated according to the editing instruction, the fifth column is the mask of the editing area, and the last column is the final generated image. A total of 7 sets of experimental results are listed. The first group is the adding task, which provides the original image, editing instructions (holding a bag in the right hand), and the reference image (bag), and finally generates an image of a person holding the bag; the second group is the adding task, which provides the original image, the reference image (scarf), and the editing instruction (put on the scarf), and finally generates an image of a person wearing the scarf; the third group is the editing task, which provides the original image, the editing instruction (the pants become red), and finally generates an original image with the pants changed in color; the fourth group is the editing task, which provides the original image, the editing instruction (the belt becomes pure black), and finally generates an original image of a black belt; the fifth group is the deleting task, which provides the original image (a woman wearing glasses), the editing instruction (taking off the glasses), and finally generates an image of a person taking off the glasses; the sixth group is the replacing task, which provides the original image, the reference image (another style of pants), and the editing instruction (changing the pants), and finally generates an original image with the pants replaced by the reference image; the seventh group is the replacing task, which provides the original image, the reference image (top), and the editing instruction (changing the top), and finally generates the original image with the top changed.

Claims

1. A command-driven personalized fashion image editing method, characterized in that: The steps include: Step (1), defining the categories of editing tasks, and constructing a four-dimensional data set of "original image - reference image - target image - text editing instruction" by using web crawler technology and existing image generation technology for different editing tasks; Step (2), constructing and training a target semantic network; using the text editing instructions, the reference image and the original image as inputs of the target semantic network, extracting the semantic information of the target image as a training target, obtaining the semantic information of the target image following the text editing instructions, and using the semantic information of the target image as the human body semantic information of the editing model; Step (3), build a unified editing network; add three changes based on the pre-trained instruction-driven generation model; First, a semantic control network is constructed to achieve effective guidance based on the semantic information of the target image obtained in step (2); second, a visual joint module is added to reconstruct the reference image in the editing result; third, a low-rank fine-tuning module is applied to align the understanding and implementation of the editing intention to the low-rank fine-tuning module through a small amount of training; thus, multiple editing tasks can effectively obtain corresponding editing capabilities using the same framework; Step (4): construct a multi-task low-rank fine-tuning module; through joint training, enable the framework to have the ability to align different text editing instructions to different editing tasks; finally, realize an independent yet unified fashion editing framework between different tasks.

2. The instruction-driven personalized fashion image editing method according to claim 1, characterized in that The categories of the defined editing tasks described in step (1) are add, delete, replace, and modify. A large number of noisy image-text data pairs were obtained through web crawler technology and various existing advanced image synthesis technologies. After manual screening, 36,312 pairs of edited image-text data pairs were finally obtained, of which 10,244 were replacement tasks, 10,072 were modification tasks, and 7,998 were add and delete tasks respectively. From the category point of view, ADCR involves four basic editing tasks, including 8 categories of items: tops, bottoms, full clothes, shoes, glasses, belts, bags, and scarves. The final dataset consists of the original images. , reference image , text editing commands , target image The target image is composed of a four-tuple; the original image and the reference image are both obtained by the crawler, and the text editing instructions are generated by the large language model based on the comparison between the original image and the target image; the acquisition of the target image is concentrated in the following situations: for adding or deleting tasks, the original image containing the reference image is filtered, and the reference image is removed through SD1.5-Inpaint; the target image in the replacement task uses the existing virtual fitting model; the target image in the modification task is modified by the editing model and then synthesized using the virtual fitting model.

3. The instruction-driven personalized fashion image editing method according to claim 1, characterized in that The human semantic information described in step (2) includes several human region labels, and the number of human semantic labels is 18, including background, hat, hair, sunglasses, top, skirt, pants, dress, belt, left shoe, right shoe, face, left leg, right leg, left arm, right arm, bag, and scarf regions, which effectively represent the structural information of the human body and related clothing. The target semantic network is essentially a graph-generated graph model, which uses the stable diffusion model as the basic model and uses the original image in the data pair as the basic model input after forward noise addition. The target semantic network serves as the human semantic guidance prior of the editing model and needs to generate images that follow the text editing instructions. And the target semantic image conforms to the original human semantics ; The image feature extractor CLIP is used to obtain the 24-layer feature vector of the reference image, i.e., the image embedding. The first 6 layers of feature vectors representing the low-frequency information of the reference image are averaged and then passed through a set of The constructed mapping network is converted to the text embedding space and then mixed with the text embedding to obtain the final text editing instructions containing image features. , thus completing the hybrid embedding; The training loss adds a regularization constraint on the feature vector based on the diffusion loss. The specific formula is as follows: ； in, is the VAE variational autoencoder in the stable diffusion model, Represents the latent variable obtained by encoding the input image x through the variational autoencoder , represents the noise distribution extracted from image x, represents the time step, is noise randomly sampled from a normal distribution, Is a function that converts the text template Embedded with images Combined, generate hybrid embeddings for guiding text generation; here represents the final text editing instruction, and is the image embedding, which represents the features extracted from the image; is a hyperparameter used to adjust the strength of the eigenvector regularization constraint; Represents the use of image feature extractor CLIP from the reference image The nth layer feature vector extracted from is a reference image used to provide visual information for training in conjunction with text, Represents a multilayer perceptron.

4. The instruction-driven personalized fashion image editing method according to claim 1, characterized in that The unified editing network described in step (3) is specifically implemented as follows: The original image After being input into the unified editing network, the variational autoencoder that comes with the unified editing network Convert to original latent representation , and then randomly sample noise in the normal distribution as , the initial noise With the original latent representation Concatenate in the channel dimension to obtain potential representation ; The original image passes through the target semantic network to obtain the target semantic image , the semantic control network transforms the target semantic image After being converted into semantic latent representation, a sequence of semantic feature vectors is extracted , and serves as an input to the unified editing network; After the reference image features are extracted by the image feature extractor CLIP, they are input into the visual joint module of the unified editing network; Latent representation of reference image features in the cross-attention layer of the unified editing network and the editing process For each different editing task, a low-rank fine-tuning module is applied to each attention layer in the unified editing network, and finally Step denoising and decoding of the variational autoencoder, the potential representation In the unified editing network The latent representation obtained after step denoising , and decode the output as the edited image .

5. The instruction-driven personalized fashion image editing method according to claim 4, characterized in that The semantic control network uses the encoder part of the generative network as the basic structure, and the initial weights are copies of the corresponding generative network weights; Semantic Control Network uses target semantic image As input, take the final text editing command As a text guide; the entire semantic control network is regarded as a semantic information feature extractor, and each time step Each layer Potential manifestations of As the target semantic feature, after being mapped by the corresponding zero convolution layer of each layer, it is used as the corresponding potential representation in the decoder of the residual and generative network Addition guides the generation of target images that conform to text instructions; Use the stable diffusion model as the generating network to transform the target image Through variational autoencoder Transform into target potential representation And add noise as the input of the generating network The textual guidance of the denoising process is the target image description generated by a multimodal large language model ; The semantic control network transforms the target semantic image After being converted into semantic latent representation, a sequence of semantic feature vectors is extracted , and at zero convolution After mapping, the residual is added to the corresponding potential representation layer by layer in the decoder of the generative network to provide semantic guidance for the editing process; the training process freezes all parameters except the semantic control network and only updates all parameters of the semantic control network; the training loss adopts diffusion loss, and the formula is defined as follows: ； in, Represents the potential representation of the input image x after being transformed by the variational encoder (VAE) ; Represents the description of the target image; is the noise sampled from a standard normal distribution, and t represents the time step in the diffusion process; Represents the target semantic image.

6. The instruction-driven personalized fashion image editing method according to claim 4, characterized in that The visual joint module is an augmented network of the basic cross-attention layer; first, the cross-attention layer in the unified editing network is defined as follows: ； in, Represent the original query features, key features, and value features respectively; refer to Design derived from text embedding; visual joint module proposes a learnable weight matrix and , the reference image extracted by the visual encoder CLIP Embedding maps to visual key features and visual value features ; Therefore, the hybrid cross attention layer augmented by the visual joint module is defined as follows: ； in, Dimensions representing key and query features, and Represents the key and value features introduced by the visual joint module, which are composed of the reference image Visual features and weight matrix extracted by visual encoder and Multiply them together to get .

7. The instruction-driven personalized fashion image editing method according to claim 4, characterized in that: Apply the low-rank fine-tuning module to the mapping layer, self-attention layer, and cross-attention layer in each attention layer of the unified editing network; The low-rank fine-tuning module adds a low-rank residual network to all linear layers and convolutional layers. Its definition can be abstracted as the following formula: ； in, represents the forward computation of any linear layer or convolutional layer, is the input image of the unified editing network; represents the weight of the low-rank fine-tuning module; Represents the weight of the original linear layer or convolutional layer, and its dimension is ; represents the low-rank fine-tuning module, constitute, and The corresponding dimensions are and Linear or convolutional layers, is the rank of the low-rank fine-tuning module; Fine-tune the corresponding unified editing network for different editing tasks to obtain a set of low-rank fine-tuning modules corresponding to the editing tasks; freeze all parameters except the low-rank residual network during training, that is, only keep the parameter updates of the low-rank residual network; training loss Including diffusion loss and editing positioning loss, the formula is defined as follows: ； Formula (8) represents Diffusion loss, where represents the unified editing network after optimization by the low-rank fine-tuning module; Represents the input image After the variational autoencoder The transformed potential representation z; A text description representing the target image; represents the noise sampled from the standard normal distribution; t represents the time step in the diffusion, which controls the progress of the generation process; represents the noisy latent representation at time step t, which serves as the input of the generative network; A latent representation representing the original image; represents the reference image, which is used as a condition for the denoising process; Represents the target semantic image; The edit localization loss expressed in formula (9) ; In the single-step denoising process, each layer of the cross attention map is defined as , that is, Layer The attention map corresponding to the word unit, the total number of word units is ; Edit the localization loss It is Layer The average attention map of the word unit; because the sizes of attention maps of different layers are not all the same, according to experience, the average attention map The size is uniformly scaled to 16*16 resolution; edit the positioning loss Represents the edit area mask, with a resolution of 16*16. The mask data is an augmentation of the basic dataset and is obtained through the SAM segmentation model.

8. The instruction-driven personalized fashion image editing method according to claim 7, characterized in that The multi-task low-rank fine-tuning module in step (4) is essentially an MLP multi-layer perceptron for classification tasks, using text editing instructions The corresponding text embedding is used as input, the feature dimension of the text embedding is aligned with the dimension of the number of tasks, and the output of the multi-task low-rank fine-tuning module is called the low-rank weight ; The low-rank fine-tuning module set obtained by independent editing task training is weighted and merged, and the low-rank weights obtained by mapping As the activation degree of the low-rank fine-tuning module of the corresponding task; therefore, the abstract definition of the low-rank fine-tuning module in formula (6) is updated to the following formula: ； represents the output generated by the unified editing network; represents the initial linear transformation matrix or weight matrix, which is applied to the input image ; represents the low-rank weight of the i-th low-rank fine-tuning module, which is the result of the output of the low-rank fine-tuning module SEL; represents the weight change of the i-th low-rank fine-tuning module, that is, the deviation part generated by the low-rank fine-tuning process; , represents the matrix decomposition part of the i-th low-rank fine-tuning module; Among them, the low-rank weight , Represents 4 groups of independently trained low-rank fine-tuning modules; Finally, a joint training strategy is designed to perform end-to-end self-supervised training on the multi-task low-rank fine-tuning module; the training samples used in the joint training are randomly sampled from the "original image-reference image-target image-text editing instruction" four-dimensional data set; the diffusion loss is maintained on the basis of the classification loss, and the specific selection loss The definition is as follows: ； in, Similar to formula (8), but the merged low-rank fine-tuning module calculates the update as formula (10); represents the same text feature extraction module as the unified editing network, namely CLIP; It is a one-hot encoding related to the editing task of the training sample, which ensures that the multi-task low-rank fine-tuning module activates the correct low-rank fine-tuning module while minimizing the activation of the wrong editing ability; represents the low-rank fine-tuning module; ； represents the diffusion loss, where represents the unified editing network after optimization by the low-rank fine-tuning module; Represents the input image After the variational autoencoder The transformed potential representation z; A text description representing the target image; represents the noise sampled from the standard normal distribution; t represents the time step in the diffusion, which controls the progress of the generation process; represents the noisy latent representation at time step t, which serves as the input of the generative network; A latent representation representing the original image; represents the reference image, which is used as a condition for the denoising process; Represents the target semantic image.

Images