Clothing accessory local replacement method, medium and electronic equipment

Abstract

The invention provides a clothing accessory local replacement method, a medium and electronic equipment, and belongs to the technical field of computers and networks, the method comprises the following steps: constructing weak supervision triple data by using SDXL-informing and SAM models, and enhancing spatial positioning and boundary learning; a PositionIC synthesis strategy is introduced to expand paired data to construct consistent triple data, and scale robustness and position signal perception are improved. In training, the MOE-C-LoRA realizes differentiated learning of multi-modal features through an expert branch and router mechanism, and realizes feature decoupling and accurate fusion in combination with conditional multi-modal DiT attention and an attention condition position mask. The training adopts two-stage combined loss: in the first stage, de-noising is emphasized, the background is kept balanced with experts, and position accuracy and boundary smoothness are ensured; in the second stage, consistency and perception constraints are emphasized, and the appearance and visual reality of the accessories are improved. The method can be widely applied to scenes such as virtual fitting, e-commerce display, digital human generation and costume design, and has high application value and popularization potential.

Description

Technical Field

[0001] This invention belongs to the field of computer and network technology, and specifically refers to a method, medium, and electronic device for partial replacement of clothing accessories. Background Technology

[0002] With the rapid development of e-commerce and virtual try-on technology, consumers are no longer satisfied with the traditional "whole garment change" try-on experience. Instead, they want to accurately and naturally replace specific clothing accessories (such as glasses, earrings, belts, necklaces, etc.) on the same model to achieve highly personalized and scenario-based outfit previews. However, existing technologies still face three major bottlenecks in this task: First, the coupling of multiple conditions is difficult. Traditional methods (such as ControlNet and IP-Adapter) usually only support a single control signal (such as text or mask), making it difficult to handle multimodal inputs that combine text prompts, accessory images, and precise location masks simultaneously, resulting in inconsistencies between local replacement areas and global styles. Second, there are conflicts between elements and positions. Existing UNet-based diffusion models lack fine-grained spatial control capabilities, which can easily lead to problems such as accessory misalignment, element drift, or interference between adjacent areas, especially when multiple accessories are replaced simultaneously. Finally, there is a strong data dependency. High-quality "model-accessory-mask-result" quadruple data required for model training is scarce. Traditional supervised methods require a large number of paired training samples, making it difficult to cover the diverse outfit needs in real-world scenarios. In recent years, the rise of Diffusion Transformer (DiT) has provided a systematic solution to this bottleneck, driving the task of partial clothing replacement from the laboratory to the entry point of the clothing industry, and providing key traffic and business opportunities for platforms such as live e-commerce and fast fashion design. Summary of the Invention

[0003] In order to solve the problems existing in the prior art, the present invention proposes a method, medium and electronic device for partial replacement of clothing accessories.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] On one hand, the present invention provides a method for partial replacement of clothing accessories, comprising the following steps:

[0006] S1: Obtain paired images of clothing models with accessories and images of clothing models without accessories from the database. There are 5,000 pairs of images of clothing models with accessories, each containing an image of an accessory and a corresponding image of a model with accessories. There are 20,000 images of clothing models without accessories. Use a large language model to generate text descriptions for the selected images (including: model body proportions, accessory elements, clothing features, etc.). Use 30,000 images and 25,000 text description prompts together as training data.

[0007] S2: Users manually upload images of clothing models for which partial replacement of clothing accessories is required, as well as images of accessories to be replaced and corresponding text descriptions (the number of accessory images entered by the user should not exceed 3), and save them into the system after preprocessing for subsequent operations.

[0008] S3: For 20,000 clothing model images without paired accessories, a pre-trained SAM model is used to locate the corresponding accessory positions and generate an accessory conditional mask. Simultaneously, based on this mask, a pre-trained SDXL-inpainting model is used to perform a slight diffusion repaint of the image-generated region to remove identifiable accessory traces. This is denoted as a clothing model image with blurred accessory boundaries. This image, along with the original clothing model image with accessories and the fully masked accessory conditional mask (pure black mask), forms a weakly supervised triplet data pair. To improve the model's ability to generalize to natural images, a PositionIC synthesis strategy was used. In 5000 pairs of images of clothing models with accessories, individual accessory samples were randomly placed on different boundaries or angles to synthesize training data pairs, generating "synthetic pairings" with positional signals to augment the small number of paired data. The SAM model was then applied to the clothing model images to generate positional mask maps of the accessories. After augmentation, the total number of pairs of clothing model images with accessories was 10000. The consistency triples formed by the clothing model images with accessories and their accessory mask maps are denoted as... This is used to increase location diversity and scale robustness;

[0009] S4: The Flux-based Diffusion in Transformer (DiT) architecture uses a two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) for secondary training. In the first stage, weakly supervised triplet data is used to enable the model to learn the spatial location information and boundary fusion information of the accessories. In the second stage, consistent triplet data is used to enable the model to learn to generate accessories with consistent appearance in the specified area. That is, the target localization is first learned with unpaired data, and then the target consistency is learned with a small amount of paired data.

[0010] S5: During training, the Hybrid Expert Conditional LoRA Fine-Tuning Strategy (MOE-C-LoRA) abandons the traditional idea of ​​grafting low-rank matrix training onto a few specific layers. It innovatively introduces the Conditional Multimodal DiT Attention method (CMMDiTA) and the Attention Conditional Position Masking mechanism (ACPM) in the low-rank matrix training of all experts. The Conditional Multimodal DiT Attention method is to keep each conditional branch with its own receptive field to avoid mutual interference between multiple conditions, so as to efficiently expand between multiple conditions and single conditional inputs. The Attention Conditional Position Masking mechanism enables the model to more clearly divide the appearance texture information and geometric position information in the feature space to accurately achieve local replacement, reduce false coverage and visual artifacts, and thus be more robust when repeating the same accessory in different positions or sharing appearance information between different accessories.

[0011] S6: A joint loss function is used to train the parameters and backpropagate the two-stage hybrid expert conditional LoRA fine-tuning strategy. This loss consists of diffusion denoising loss, consistency loss, background preservation loss, expert load balancing loss and pixel perception loss, which are combined and applied in the two-stage training process to optimize the entire diffusion process.

[0012] S7: Load the model weights and network structure of the multi-condition hybrid expert LoRA and DiT trained in S6, fuse the expert LoRA and DiT basic models, obtain the clothing model image and accessory image that need to be locally replaced and the corresponding text description from the user input from the system, and apply a mask to the clothing model image to clarify the spatial area to be replaced by the accessory image, and then input it into the loaded DiT method.

[0013] S8: During the inference process, the user-input clothing model image, accessory image, and drawn mask image are first fed into the VAE module to extract the main image features, accessory image features, and positional encoding features, respectively. Then, the T5 text encoder module extracts the text features of the user-input text description. The extracted main image features, accessory image features, and positional encoding features are then uniformly encoded. The encoded accessory image features and positional encoding features are added together to form the final conditional features, which are then input into the DiT architecture to gradually estimate the latent space feature results of the denoised version of the clothing model image. Finally, the VAE is used to decode and output the corresponding clothing model image with accessory replacement results.

[0014] Optionally, the specific steps for obtaining clothing image data in step S1 are as follows:

[0015] S11: Filter images containing the category terms "accessories" and "model" from the clothing database, and match them by image ID. The successfully matched image groups are divided into 5,000 pairs of clothing model images with accessories, and the unmatched clothing model images with accessories are divided into 20,000 clothing model images without accessories.

[0016] S12: Store the paired images of clothing models with accessories and the images of clothing models without accessories separately for the subsequent data preprocessing steps.

[0017] Optionally, the method for preprocessing the dataset in step S3 is as follows:

[0018] S31: Select 20,000 clothing model images without paired accessories, use the pre-trained SAM model to locate the spatial position of the accessories on the model images, and generate an accessory conditional mask image.

[0019] S32: Use the accessory conditional mask map to guide the SDXL-inpainting model to perform local micro-diffusion repainting of the accessory area of ​​the clothing model image to remove identifiable accessory traces. The output result is recorded as a clothing model image with blurred accessory boundaries.

[0020] S33: Combine and pair the original clothing model image with accessories before processing by the SDXL-inpainting model, the generated clothing model image with blurred accessory boundaries, and the fully masked accessory conditional mask image to form 20,000 weakly supervised triplet data pairs. ;

[0021] S34: Based on the PositionIC synthesis strategy, data augmentation was performed on 5000 sets of clothing model images with accessories. The accessory image samples were randomly placed on different boundaries and angles to form synthetic training data pairs with the original clothing model images. After the augmentation, the total data volume of clothing model images with accessories was 10000 pairs.

[0022] S35: In the paired group of clothing mannequin images with accessories, the SAM model is used to segment and locate the accessory positions, obtaining the accessory position mask. The expanded clothing mannequin images with accessories and their accessory mask are combined to form a consistency triplet. This enables the expansion of paired image data;

[0023] S36: By introducing the large-scale multimodal language model Qwen2.5 (VL-32B-Instruct), the clothing model images in the consistency triplet and weakly supervised triplet are labeled respectively. The large model is required to describe the key elements such as the model's dressing style, the position of accessories, accessory elements, and clothing features in as much detail as possible, and obtain a total of 30,000 corresponding text description prompts.

[0024] S37: 20,000 weakly supervised triplet pairs are used for the first stage training of a two-stage hybrid expert-conditional LoRA fine-tuning strategy to improve the model's blind target localization ability and form priors for spatial localization and synthesis. 10,000 consistency triplet pairs are used for the second stage training to improve the consistency of the model's accessory texture, ensuring high consistency between the replaced accessory and the target reference. Finally, the text prompts and image data are integrated to obtain the input data for the first stage, represented as follows: The input data for the second stage is represented as The dataset was divided into training, validation, and test sets in a ratio of 8:1:1.

[0025] Optionally, the method for LoRA training using the two-stage hybrid expert-conditional LoRA fine-tuning strategy in step S5 is as follows:

[0026] To effectively extract and adapt the texture features and spatial positions of accessories in the task of partial replacement of clothing accessories, this invention introduces a two-stage hybrid expert-conditional LoRA fine-tuning strategy to perform secondary fine-tuning training on the Flux model. For this purpose, this invention adjusts the weights of arbitrary linear layers in the DiT structure. , Introduction LoRA, one expert, each expert It is responsible for learning different key feature elements, such as spatial location features, accessory texture features, clothing style features, and boundary blending features. Its low-rank correction parameter is... and The scaling factor is denoted as Meanwhile, a conditional router is defined. For a given conditional vector, a small multilayer perceptron (MLP) outputs the corresponding softmax weights, which are then applied to subsequent attention calculations and linear transformations. In addition, expert load constraints are defined to avoid route collapse. LoRA injection correction is used on DiT to avoid updating a large number of weight parameters. Combining conditional routing allows different experts to learn specialties such as "location / texture / material / style", and achieves a good trade-off between parameters, training efficiency and conditional flexibility.

[0027] For the LoRA training process, this invention innovatively divides it into two stages. In the first stage, a spatial localization expert (one of multiple hybrid expert models LoRA) and other LoRA experts learn how to redraw accessory points on unpaired images and perform natural fusion. This stage is a weakly supervised training process mainly used to improve the model's spatial localization ability. In the second stage, other expert models LoRA are trained to maintain consistency in texture and shape between the accessory mask region of the clothing model image and the input accessory image. At the same time, the Conditional Multimodal DiT Attention Method (CMMDiTA) and Attention Conditional Position Masking Mechanism (ACPM) are introduced in the training process of the above two stages to conditionally perceive and restrict the attention, enabling the model to obtain multi-conditional scalability and precise local control.

[0028] Optionally, the conditional multimodal DiT attention method (CMMDiTA) in step S5 is:

[0029] Traditional Transformer attention is global, meaning all tokens can compute attention weights with each other. However, this easily leads to interference between conditions in multi-condition input scenarios, making it difficult to distinguish between positional and semantic information, and causing the model to generate irrelevant features at incorrect locations. To address this, this invention designs a conditional multimodal DiT attention method. Its core idea is to assign independent receptive fields (attention subspaces) to different conditions, allowing for controlled interaction between global and conditional local information. This ensures that each conditional branch maintains its own receptive field during denoising and token fusion, avoiding mutual interference and effectively scaling to multiple conditional input scenarios. For feature vectors from model images, text prompts, accessory images, and mask images:

[0030]

[0031] in, It is the first The conditional encoding vector of each accessory is composed of a weighted sum of the feature vector of the accessory image and the positional features of the mask image. First, the input feature sequence... The query matrix (Q) is divided into parts derived from the main clothing model images and text. The matrix is ​​defined as a token that allows global access, ensuring overall consistency. Correspondingly, the query matrix will be based on the accessory conditions. Defined as only being able to access its corresponding local subsequence features The token is used to avoid interference between different accessory conditions. Then, multimodal attention is applied. For the global branch, the text, image subject and all conditional information features are fused. For the conditional branch, attention is calculated separately for each accessory condition to ensure that feature alignment is performed only in a local range. Finally, the results from the global branch and the conditional branch are concatenated and then fed into the next module through a linear projection layer.

[0032] Optionally, the attention conditional location masking mechanism (ACPM) in step S5 is as follows:

[0033] While CMMDiTA effectively addresses the issue of mutual interference between conditions, it only controls the interaction between different conditions and cannot control the generation of corresponding conditional features in the local feature space. Therefore, a method is still needed to allow the model to accurately determine the specific location that needs to be replaced in the spatial dimension. Typically, binary masks are used for feature space localization. However, in DiT structures primarily based on transformers, the feature vector tokens are at the patch level, making direct feature alignment with pixel-level binary masks difficult. This can easily lead to unclear encoding of accessory conditions, resulting in out-of-bounds content or visual artifacts. To address this, this invention proposes an Attention Conditional Position Masking (ACPM) mechanism. Its core idea is to align pixel-level spatial mask information to the patch-level token space and incorporate attention weights, enabling the model to distinguish between "accessory regions" and "background regions" at the attention level. First, the input mask image is pooled by a position encoder according to the downsampling rate and block size to obtain a token-level soft mask. Each token will have a corresponding soft mask, which serves as a weighting factor during attention calculation. Finally, the soft mask is subjected to Gaussian blurring to further soften the boundaries and reduce the "hard edges" of the generated result, achieving a smooth transition of the edges. This makes the generated accessories blend more naturally with the background and reduces generated artifacts.

[0034] Optionally, the method for performing two-stage model optimization using the joint loss function in step S6 is as follows:

[0035] This invention proposes a joint loss function for parameter training and backpropagation of a two-stage hybrid expert-conditional LoRA fine-tuning strategy. This loss is derived from diffusion denoising loss. Consistency loss Background retention loss Expert load balancing loss With pixel-perceptual loss The training objective of the joint loss is to use a DiT-based multi-condition generator to accurately and naturally replace the most elements on the model image, given a model image plus multiple accessory reference images, a position mask for each accessory, and optional text cues. (suggestion This invention aims to regenerate accessory elements at key accessory locations in unpaired images while maintaining consistency in texture and appearance with the integrity of the non-replaced areas of the model. To this end, the model training process is divided into two stages. In the first stage, a localization expert (LoRA) (one of multiple hybrid LoRA experts) and other LoRA experts are trained to teach the model how to regenerate accessory elements with natural integration of boundaries at key accessory locations in unpaired images, primarily through joint diffusion denoising loss. Background retention loss With expert load balancing loss In the second stage, the model is optimized by training other hybrid experts (LoRA) to ensure high consistency with the target accessory, primarily through joint diffusion denoising loss. Consistency loss Background retention loss Pixel-perceptual loss and expert load balancing loss Optimize the model to ensure stable learning of conditional image information.

[0036] Optionally, the method for fusing the model weights and network structure of the multi-condition hybrid expert LoRA and DiT trained in step S7 is as follows:

[0037] The weight fusion process involves injecting the low-rank adjustment of conditional specialization learned by LoRA into the weight matrix of the DiT backbone, thereby obtaining a model that has both global generation capability and local conditional adaptability. This not only inherits the prior knowledge in the original DiT weights, but also introduces the conditional specialization capability of LoRA, which is more conducive to applications in specific scenarios. Specifically, firstly, the pre-trained DiT backbone model weights, including the original weights of all Transformer layers, are loaded. Simultaneously, the trained MOE-C-LoRA weights are loaded, which store the corresponding low-rank decomposition matrix for each conditional expert LoRA. Then, a dynamic conditional fusion strategy is adopted. Unlike the traditional method of statically adding LoRA weights to the original model weights, the proposed two-stage hybrid expert conditional LoRA fine-tuning strategy calculates dynamic gating coefficients based on input conditional features during inference, and then the router dynamically selects and weights the expert LoRAs. Subsequently, the fused weights are replaced in the corresponding layers of the DiT model, and the fused overall weights are saved as new model checkpoints for inference. Finally, the model network results and the model weights with fused LoRA expert parameters are loaded. Under the action of the CMMDiTA and ACPM modules, the model uses the weights to calculate attention and feature updates, gradually spreading the generation of results.

[0038] Optionally, the method for inference using the fused DiT based on the user-input clothing model image, accessory image, custom mask image, and text description in step S7 is as follows:

[0039] S71: First, the system requires users to upload images of clothing models and accessories. Then, the system applies a custom mask to the clothing model images to determine the area on the upper body of the accessories and obtains the mask image. At the same time, the system performs preprocessing operations on the input clothing model images, accessory images, and mask images to ensure that the image data is suitable for subsequent model calculations. In addition, the relevant text descriptions entered by the user will also be transferred to the system for later use.

[0040] S72: The preprocessed image and text description are combined and input in parallel into the Flux model with weight correction applied by MOE-C-LoRA. The CMMDiTA masking mechanism is used in the attention layer. First, a VAE encoder maps the clothing model image to latent space noise. Then, a CLIP visual encoder is used to extract the image feature vector of the accessory image. Subsequently, a position encoder is used to encode the mask image into a position vector. Next, a T5 text encoder converts the text description into a semantic feature vector. Finally, the image feature vector and the position encoded vector are weighted and added together, which makes it easier to effectively control the noise influence range and accessory appearance features in subsequent attention calculations.

[0041] S73: Input the features obtained in S72 above into the DiT architecture. The system will start from pure noise and gradually estimate the latent space representation of the denoised version of the clothing model image in the latent space. The features are continuously fine-tuned by applying different expert LoRA. In addition, the Conditional Multimodal DiT Attention Method (CMMDiTA) is used to ensure the stability of the feature space position, and the Attention Conditional Position Masking Mechanism (ACPM) is used to ensure the consistency of the accessory features.

[0042] S74: Input the latent space representation of the clothing model image obtained in S73 into the pre-trained VAE decoder module for decoding and reconstruction to obtain the final output clothing model image. This model image contains a high-quality image with clear edge blending, texture structure and consistent appearance after the target accessory is replaced in the specified location area, thus realizing the user-specified target accessory replacement task.

[0043] S75: The system outputs the corresponding user-input-generated clothing model image with target accessories to the user interface for real-time preview, and can export and save it as JPG, PNG or PDF formats, improving the image's secondary editing capabilities and application flexibility.

[0044] On the other hand, the present invention also provides an electronic device for partial replacement of clothing accessories, comprising: one or more processors;

[0045] A storage system for storing one or more programs that, when executed by one or more processors, enable the one or more processors to implement a method for generating partial replacements of clothing accessories.

[0046] On the other hand, the present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method for generating partial replacement images of clothing and accessories.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention proposes a method for local replacement of clothing accessories based on the Diffusion Transformer (DiT) architecture and a two-stage hybrid expert condition LoRA fine-tuning strategy (MOE-C-LoRA). It achieves significant performance improvement in the process of local replacement of clothing accessories and effectively overcomes the three core bottlenecks of existing technologies: multi-condition coupling, element position conflict and strong data dependency.

[0049] Firstly, regarding multi-condition processing capabilities, this invention innovatively introduces the Conditional Multimodal DiT Attention Method (CMMDiTA), which allocates independent attention subspaces for different input conditions such as text prompts, accessory images, and location masks, avoiding mutual interference between multimodal information. Compared to the limitations of traditional ControlNet and IP-Adapter, which only support single control condition signals, this solution can simultaneously and accurately process multimodal and multi-condition inputs, ensuring a high degree of consistency between the local replacement area and the overall clothing style and color tone of the model. This solves the problem of "disconnect between accessories and overall outfit" in existing technologies. For example, when replacing a model's necklace and earrings, it can simultaneously match the texture of the clothing material and the lighting style of the wearing scene.

[0050] Then, in terms of spatial positioning accuracy, the attention-conditional position masking (ACPM) mechanism is used to achieve precise alignment between pixel-level masks and block-level token features. This mechanism pools the mask image into a token-level soft mask according to the model downsampling rate, incorporates an attention weighting factor, and combines Gaussian blur to optimize the boundary transition effect, effectively solving the accessory misalignment and element drift problems that are prone to occur in existing diffusion models.

[0051] Secondly, regarding data utilization efficiency, a dual data system of weakly supervised triples and consistency triples is constructed, significantly reducing the reliance on high-quality paired data. On one hand, using the pre-trained SAM model and SDXL-inpainting model, 20,000 unpaired accessory images are transformed into weakly supervised data with position masks, allowing the model to autonomously learn accessory localization and boundary fusion. On the other hand, the PositionIC synthesis strategy expands the original 5,000 pairs of paired data to 10,000 pairs, combining a small amount of real data to improve positional diversity and scale robustness. Compared to traditional supervised learning, which requires tens of thousands of "model-accessory-mask-result" quadruples of data, this solution can achieve generalized adaptation of natural scene images based on only 5,000 pairs of original paired data, reducing data acquisition costs by more than 60%.

[0052] Finally, regarding model training efficiency and generalization ability, the two-stage MOE-C-LoRA fine-tuning strategy achieves efficient learning through "localization first, then consistency." The first stage uses weakly supervised data to train spatial localization experts, enabling the model to master accessory region recognition and background fusion logic. The second stage uses consistency data to train texture and style experts, ensuring a high degree of consistency between the generated accessories and the reference image. This phased training approach avoids the computational cost of full parameter fine-tuning (only requiring updates to the expert LoRA low-rank matrix, reducing the total number of parameters by 85%), and through a dynamic conditional routing mechanism, allows different experts to specialize in tasks such as "spatial localization," "texture restoration," and "style matching." This ensures stable performance across different body types and accessory types (such as metal necklaces, fabric belts, and framed glasses), with scale robustness improved by 53% compared to existing DiT base models.

[0053] This invention, through an innovative method for partial replacement of clothing accessories, achieves for the first time end-to-end automated generation from a precise image of clothing accessories worn on a model to a specific part of the garment, while simultaneously ensuring stylistic diversity, structural accuracy, and controllable craftsmanship details. The generated results not only reproduce the texture details of the accessories (such as chain weaving patterns and gemstone cutting luster), but also allow for adjustments to the wearing status of the accessories based on text prompts (such as "belt knot looseness" and "necklace pendant angle"), meeting personalized outfit preview needs, enhancing the online shopping experience for consumers, and thus promoting sales and market competitiveness. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method for generating partial replacement of clothing accessories according to the present invention;

[0055] Figure 2 This is a diagram of the algorithm structure based on the Flux-DiT architecture of the present invention;

[0056] Figure 3 This is a flowchart of the first-stage training algorithm in the MOE-C-LoRA of this invention;

[0057] Figure 4 This is a flowchart of the second-stage training algorithm in the MOE-C-LoRA of this invention;

[0058] Figure 5 This is a structural diagram of the conditional multimodal DiT attention method of the present invention;

[0059] Figure 6 This is a diagram showing the result of partial replacement of clothing accessories according to the present invention. Detailed Implementation

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0061] like Figure 1-6 As shown in the embodiment, a method for partial replacement of clothing accessories is provided.

[0062] Includes the following steps:

[0063] (1) First, image data categorized under "accessories" and "model" were retrieved from the database. These images were then paired using their image IDs, resulting in 5000 pairs of images featuring models wearing accessories and 20000 images featuring models without accessories. Among these, the pairs of images featuring models wearing accessories... Data augmentation was performed using the PositionIC synthesis strategy to expand the number of pairs to 10,000, and the expanded clothing model images were then analyzed using the SAM model. Automatically locate and segment the positions of accessories to obtain a positional mask image of the accessories. Images of clothing models and accessories The combination of its accessory mask image and the mask image is denoted as a consistency triple. The clothing model image without paired accessories is redrawn by locally applying a small amount of diffusion to the clothing accessory image area using the SDXL-inpainting model and the SAM model, resulting in a clothing model image with blurred accessory boundaries. Remove identifiable accessory traces and replace the original unmatched accessory clothing model image. The blurred clothing model image generated after inpainting With full mask image Together they form a weak supervision triplet Next, the large-scale multimodal language model Qianwen 2.5 is introduced to label the clothing model images in the consistency triplet and weakly supervised triplet respectively. The labeling is required to describe the model's clothing style, accessory placement, accessory elements, clothing features, and other key elements in as much detail as possible, resulting in a total of 30,000 corresponding textual description prompts. Finally, the textual prompts and image data are integrated to obtain the input data for the first stage. The input data for the second stage is represented as .

[0064] (2) Users manually upload images of clothing models that need to be partially replaced with accessories, images of accessory conditions (no more than 3 conditions), and text prompts describing the accessory replacement in the system. They also manually draw the corresponding accessory replacement area mask on the clothing model image. It is worth noting that as many mask areas are required for each set of conditions entered. Otherwise, multiple condition accessories will be generated under the same mask area. After preprocessing, the images are entered into the system and saved for subsequent operations.

[0065] (3) The method for local image repainting using the SDXL-inpainting model is as follows:

[0066] First, regarding the unpaired clothing model image dataset The pre-trained SAM model is used to automatically locate and segment potential accessory regions, obtaining a binary location mask image. ,in , These represent the height and width of the input image, respectively. Then, the obtained mask image... Its original clothing model image Together, they are input into the pre-trained SDXL-inpainting model for local diffusion-based inpainting of the original image. The mathematical expression is as follows:

[0067]

[0068] in, This represents the repainting operator for the SDXL-inpainting model. This represents random noise in the diffusion model. Through this diffusion process, the model regenerates and fills in the pixels of the masked region, while keeping the non-masked region consistent with the original image, thus obtaining a model image with smooth boundaries and weakened accessory traces. Finally, the original model image... The full mask of the corresponding mask image Model image with blurred boundaries after inpainting processing Forming a weakly supervised triplet:

[0069]

[0070] This triple will be used in subsequent training phases to enhance the model's ability to distinguish between accessory positions and background boundaries, enabling the model to learn the spatial distribution patterns of accessory images in clothing model images without paired labeled data.

[0071] (4) The method for data augmentation based on the PositionIC synthesis strategy is as follows:

[0072] A dataset of paired clothing and accessories model images for a small sample size This invention employs a PositionIC location and identity consistency constraint synthesis strategy to expand the data, thereby improving the model's location signal perception capability and scale robustness. First, for the input accessory image... By employing affine transformation and random sampling strategies, accessory images are applied to model images. The position, angle, and scale are perturbed, and then the PositionIC operator is used to obtain the synthetic sample:

[0073]

[0074] in, This represents random affine transformation operators (including rotation, scaling, pinyin, and boundary clipping). This indicates the use of the SAM model in the original clothing model image. The accessory location mask is obtained from the image. Then, the SAM model is used to synthesize the model image. Automatic identification and mask extraction of accessory positions are performed to obtain the updated spatial constraint mask. To obtain consistent triples, ensure that the position of accessories in the image aligns with the geometric structure.

[0075]

[0076] During training, the PositionIC-based synthetic data was used alongside the original real-world pairing data. This enabled the model to learn the generation patterns of accessories at different locations and scales, while maintaining identity consistency and enhancing its spatial generalization ability. Finally, data augmentation expanded the original 5,000 accessory pairs to 10,000 pairs, and divided them into training, validation, and test sets in an 8:1:1 ratio. This effectively alleviated the problem of insufficient small-sample pairing data and improved the model's robustness and adaptability in complex scenarios.

[0077] (5) The expert settings and route definition methods in the two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) are as follows:

[0078] In this invention, the hybrid expert conditional LoRA aims to perform differentiated modeling for multimodal and multi-conditional input types. Its core idea is to design the LoRA module as multiple expert branches, each focusing on feature extraction and adaptation for different condition types. Weights are dynamically allocated through a routing mechanism to fuse the experts' outputs, thereby enhancing the model's robustness and condition adaptability. For the hybrid expert setup, the linear weight matrix of a specific layer in the Transformer structure is first defined as follows: Then, introduce it into the MOE-C-LoRA strategy. Each expert The original weights are optimized using low-rank matrix factorization, mathematically expressed as follows:

[0079]

[0080] in, Indicates the first The parameter update matrix of each expert The rank hyperparameter of LoRA is usually 64 or 128. Depending on the type of input conditions, experts can be roughly divided into the following categories: appearance experts, location experts, text experts, fusion experts, texture experts, etc.

[0081] To dynamically select the appropriate expert under different input conditions, this invention relates to a conditional feature-based router function, which, for a given input conditional feature vector... The softmax weights are output using an MLP, and the mathematical formula is as follows:

[0082]

[0083] in, It is expressed as a temperature coefficient, used to assess the smoothness of equilibrium distribution. To satisfy the probability distribution constraints for each expert gating weight, the final fused weight update is as follows:

[0084]

[0085] Input conditional features Based on the gating coefficients output by the router, different experts are adaptively activated, and the model parameters are updated linearly with weights. The updated LoRA weights after fusion can be denoted as:

[0086]

[0087] in, The hyperparameter represents the weight scaling factor, used to mitigate the impact of LoRA updates on the original weights. This represents the conditional weights of a specific layer in the fused Transformer structure, used for attention and linear transformation calculations during the forward pass.

[0088] It is worth noting that not all expert models will fully participate in the two-stage fine-tuning training process. In the first stage, for the input of the weakly supervised triples, the primary focus is on activating the position expert to learn the geometric position awareness and boundary smoothing of the accessories. In the second stage, for the input of the consistency triples, the primary focus is on activating other experts such as the fusion expert, text expert, and texture expert to jointly learn the appearance consistency and semantic alignment of the accessories. Through this phased expert routing training method, the model can correctly learn the spatial position of the accessories on the clothing model and the consistency features such as the texture, appearance, and edge fusion information of the accessories, thereby achieving decoupled learning of spatial awareness and texture feature consistency.

[0089] (6) The conditional multimodal DiT attention method (CMMDiTA) in the two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) is as follows:

[0090] In the conditional multimodal DiT attention method, this invention defines two types of query branches: global branches and conditional branches, for the input token sequence. If query matrix The source is or Then a global branch is used to process the entire sequence. The key AND value Attention calculations are performed to fuse global features; if the query matrix... The source is a certain condition Then conditional branching will be used to select the key. AND value The scope is limited to local subsequences In order to avoid mutual interference between different conditions, the attention calculation method can be expressed as follows:

[0091]

[0092] in, For corresponding conditions The binary mask is used, and the positions that are off-limits are set to 0 to ensure that each conditional branch maintains its own receptive field during token feature fusion in the denoising process. It is worth noting that for each condition, two query matrices will be constructed. :

[0093]

[0094] The two outputs are obtained after attention calculation. and Fusion is performed before entering the MLP:

[0095]

[0096] By decoupling the operation of the dual-head query matrix, texture information and geometric position information can be processed more clearly, making it more robust when repeating the same accessory in different positions or sharing texture information between different accessories.

[0097] (7) The attention conditional location masking mechanism (ACPM) in the two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) is as follows:

[0098] This invention designs a special positional masking mechanism that precisely maps pixel-level positional information to the patch level in the DiT architecture, and uses this as a soft weight in the attention mechanism, prompting the model to learn how to accurately locate the specific position to be replaced in the feature space. First, for the input binary mask... This is achieved by downsampling and pooling the data to the block level using a position encoder and aligning it with the block dimension in DiT to obtain a token-level mask. The mathematical representation is as follows:

[0099]

[0100] Subsequently, the obtained token horizontal mask was Gaussian blurred. And it applies to the attention calculation of conditional branches in CMMDiTA:

[0101]

[0102] in, The weights of softmax can be regarded as an additional bias term as a constraint, which helps to restrict tokens with conditional information, so as to accurately achieve local replacement and reduce erroneous coverage and visual artifacts.

[0103] (8) The joint loss function for the two-stage training in the two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) is:

[0104] In the training process proposed in this invention, for weakly supervised triples Consistent triple data A total joint loss function was designed. This approach simultaneously constrains the denoising process of the diffusion model, the consistency of accessory texture and spatiality, and the preservation of the background of the main subject in the clothing model image, thereby achieving phased optimization of spatial awareness and semantic consistency. Firstly, for the diffusion denoising loss, similar to traditional diffusion models, its training objective is to learn the ability to reconstruct the original image from a noisy image during the diffusion process. For the input image feature vector... In the The noisy features obtained after one diffusion step are:

[0105]

[0106] At this point, the noise predicted by the model is denoted as... ,in The conditional features include text prompts, embellishment images, and corresponding positional mask images; therefore, a diffusion denoising loss is defined. for:

[0107]

[0108] This loss is primarily used to ensure that the model can accurately estimate the injected noise, thereby generating high-quality images during directional diffusion.

[0109] Secondly, regarding consistency loss The training objective is to ensure that the accessories generated in the replacement area are consistent with the reference accessories in terms of appearance and texture details. To this end, this invention defines it as follows:

[0110]

[0111] in, and These are the hyperparameter weighting coefficients. This represents the resulting image generated by the model. The present invention uses a pre-trained CNN as the feature extraction network. This loss aligns the reference and generated accessories in both pixel space and perceptual feature space, ensuring consistency between appearance and texture.

[0112] Then, to avoid unnecessary generation and redrawing of non-masked regions by the model, the present invention relates to background preservation loss. To constrain the generated image to maintain consistency with the original model image in the background, it is defined as follows:

[0113]

[0114] in, This represents the joint mask for all accessory areas. This loss ensures that only the masked areas are modified during the replacement process, while the main background and other non-masked areas remain stable.

[0115] Subsequently, to avoid an imbalance in the mixed expert capabilities caused by the frequent activation of only a few experts in MOE-C-LoRA, resulting in other experts hardly participating in training, this invention introduces an expert load balancing loss. Ensure the gating weights assigned to the router Maintain balance during training:

[0116]

[0117] in, Indicates the first The average probability of selection by each expert Indicates the first The actual number of tokens allocated to each expert, when all experts are evenly loaded. The minimum value is obtained. This loss improves the model's robustness and generalization ability by encouraging routers to utilize different experts in a balanced manner.

[0118] Next, in order to further improve the perceptual quality of the produced images, this invention innovatively designs a pixel perceptual loss. Its definition is as follows:

[0119]

[0120] in, This represents a pre-trained feature extraction network. The corresponding target image is represented by the available reference clothing model image in paired data, and the reconstructed clothing model image used for weakly supervised training in unpaired data. This loss ensures that the generated result is not only closer to the target at the pixel level, but also maintains perceptual consistency at the high-level semantic features.

[0121] Finally, regarding the two-stage training method and objectives, this invention combines the aforementioned losses to ensure they play a full and positive role in different stages. Specifically, in the first stage of training, diffusion denoising loss is primarily selected. Background retention loss With expert load balancing loss Together they form the joint loss function of the first stage. Its definition is as follows:

[0122]

[0123] in, and This is a weighted hyperparameter of the loss function, used to balance the influence between different loss functions. This loss will prompt the model to learn how to regenerate accessory elements at key accessory locations in unpaired images, allowing for a natural blending of accessory elements with boundaries; in the second stage of training, the diffusion denoising loss is primarily chosen. Consistency loss Background retention loss Expert load balancing loss With pixel-perceptual loss Together they form the second-stage joint loss function Its definition is as follows:

[0124]

[0125] in, , , and The weight hyperparameters of the loss function are used to balance the influence between different loss functions. This loss is mainly used to train the hybrid expert LoRA while ensuring the consistency of accessory features in spatial location and texture. Therefore, the overall joint loss function is defined as:

[0126]

[0127] Used to optimize network parameters throughout the two-stage training process.

[0128] (9) The two-stage training method in the two-stage hybrid expert conditional LoRA fine-tuning strategy (MOE-C-LoRA) is as follows:

[0129] In the training process of this invention, the above-mentioned overall joint loss function is used. Through phased data input and an expert dynamic activation mechanism, the spatial localization and appearance texture consistency of accessories are progressively optimized. Specifically, the first stage mainly involves weakly supervised localization training. Weakly supervised triplet data is input into the VAE encoder and text encoder respectively to obtain image features and text conditional features. Feature modeling is then performed using the DiT backbone and MOE-C-LoRA expert structure, and finally, the first-stage loss function is applied. The model output and target image are optimized to train preliminary model parameters capable of recognizing the spatial location of accessories and learning blurred boundary information. This training stage enables the model to grasp the spatial distribution patterns of accessories in the model image and its ability to smooth boundaries, laying the foundation for subsequent training on appearance and texture consistency. The second stage primarily focuses on consistency and perceptual enhancement training. By inputting consistency triplet data, and building upon the preliminary model parameters from the first stage, the second-stage loss function is applied. Further optimization was performed, while constraints were placed on diffusion denoising, consistency, and perceptual quality, prompting the model to generate results that were consistent with the reference accessory in both appearance and texture within specific regions. After multiple rounds of training, the final model parameters were obtained, which effectively combined the capabilities of position awareness and appearance / texture consistency, ensuring that the replacement result maintained a high degree of consistency with the reference accessory in both pixels and semantics.

[0130] In summary, the first stage focuses on learning "where the accessories should be placed," while the second stage focuses on learning "what the accessories should look like." The two stages of training complement each other and ultimately achieve the accuracy of local replacement and the authenticity of the generated results.

[0131] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, system, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0134] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0135] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0136] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0137] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0138] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0139] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for partial replacement of clothing accessories, characterized in that: Includes the following steps: S1: Obtain paired images of clothing models with and without accessories from the database, and use a large language model to describe the selected images with text as training data. S2: Users manually upload images of clothing models that need partial replacement of clothing accessories and images of accessories to be replaced, along with corresponding text descriptions, to the system. After preprocessing, these images are entered into the system and saved for subsequent operations. S3: For clothing model images without paired accessories, the pre-trained SAM model is used to locate the corresponding accessory positions and generate an accessory conditional mask map. Simultaneously, based on this mask, the pre-trained SDXL-inpainting model is used to perform a slight diffusion repaint of the image-generated region to remove identifiable accessory traces. This is denoted as the clothing model image with blurred accessory boundaries. This image, along with the original clothing model image with accessories and the fully masked accessory conditional mask map, forms a weakly supervised triplet data pair. To improve the model's ability to generalize to natural images, a PositionIC synthesis strategy is used. In the image of a clothing model with accessories, individual accessory samples are randomly placed on different boundaries or angles to synthesize training data pairs, generating "synthetic pairings" with positional signals to augment the small amount of paired data. The SAM model is then applied to the clothing model image to generate a positional mask map of the accessories. The consistency triple formed by the clothing model image with accessories and its accessory mask map is denoted as... This is used to increase location diversity and scale robustness; S4: The Flux-based diffusion Transformer architecture uses a two-stage hybrid expert conditional LoRA fine-tuning strategy for secondary training. In the first stage, weakly supervised triplet data is used to enable the model to learn the spatial location information and boundary fusion information of the accessories. In the second stage, consistent triplet data is used to enable the model to learn to generate accessories with consistent appearance in the specified area. That is, the target localization is first learned with unpaired data, and then the target consistency is learned with a small amount of paired data. S5: During training, the hybrid expert conditional LoRA fine-tuning strategy abandons the traditional idea of ​​grafting low-rank matrix training onto a few specific layers. It innovatively introduces the conditional multimodal DiT attention method and the attention conditional position masking mechanism into the low-rank matrix training of all experts. The conditional multimodal DiT attention method is designed to keep each conditional branch with its own receptive field to avoid mutual interference between multiple conditions, so as to efficiently expand between multiple conditions and single conditional inputs. The attention conditional position masking mechanism enables the model to more clearly divide the appearance texture information and geometric position information in the feature space to accurately achieve local replacement, reduce false coverage and visual artifacts, and thus be more robust when repeating the same accessory in different positions or sharing appearance information between different accessories. S6: A joint loss function is used to train the parameters and backpropagate the two-stage hybrid expert conditional LoRA fine-tuning strategy. This loss consists of diffusion denoising loss, consistency loss, background preservation loss, expert load balancing loss and pixel perception loss, which are combined and applied in the two-stage training process to optimize the entire diffusion process. S7: Load the model weights and network structure of the multi-condition hybrid expert LoRA and DiT trained in S6, fuse the expert LoRA and DiT basic models, obtain the clothing model image and accessory image that need to be locally replaced and the corresponding text description from the user input from the system, and apply a mask to the clothing model image to clarify the spatial area to be replaced by the accessory image, and then input it into the loaded DiT method. S8: During the inference process, the user-input clothing model image, accessory image, and drawn mask image are first fed into the VAE module to extract the main image features, accessory image features, and positional encoding features, respectively. Then, the T5 text encoder module extracts the text features of the user-input text description. The extracted main image features, accessory image features, and positional encoding features are then uniformly encoded. The encoded accessory image features and positional encoding features are added together to form the final conditional features, which are then input into the DiT architecture to gradually estimate the latent space feature results of the denoised version of the clothing model image. Finally, the VAE is used to decode and output the corresponding clothing model image with accessory replacement results.

2. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The specific steps for obtaining clothing image data in step S1 are as follows: S11: Filter images from the clothing database that contain the category terms "accessories" and "models", and match them by image ID. Successfully matched image groups are classified as paired clothing model images with accessories, and unmatched clothing model images with accessories are classified as clothing model images without paired accessories. S12: Store the paired images of clothing models with accessories and the images of clothing models without accessories separately for the subsequent data preprocessing steps.

3. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The method for preprocessing the dataset in step S3 is as follows: S31: Select a clothing model image without paired accessories, use the pre-trained SAM model to locate the spatial position of the accessories on the model image, and generate an accessory conditional mask image. S32: Use the accessory conditional mask map to guide the SDXL-inpainting model to perform local micro-diffusion repainting of the accessory area of ​​the clothing model image to remove identifiable accessory traces. The output result is recorded as a clothing model image with blurred accessory boundaries. S33: Combine and pair the original clothing model image with accessories before processing by the SDXL-inpainting model, the generated clothing model image with blurred accessory boundaries, and the fully masked accessory conditional mask image to form a weakly supervised triplet data pair. ; S34: Data augmentation of clothing model images with accessories is performed based on the PositionIC synthesis strategy. Accessory image samples are randomly placed on different boundaries and angles to form synthetic training data pairs with the original clothing model images. S35: In the paired group of clothing mannequin images with accessories, the SAM model is used to segment and locate the accessory positions, obtaining the accessory position mask. The expanded clothing mannequin images with accessories and their accessory mask are combined to form a consistency triplet. This enables the expansion of paired image data; S36: By introducing the large-scale multimodal language model Qianwen 2.5, the clothing model images in the consistency triplet and weakly supervised triplet are labeled respectively. The large model is required to describe the key elements such as the model's dressing style, the position of accessories, accessory elements, and clothing features in as much detail as possible, and obtain the corresponding text description prompt words. S37: Weakly supervised triplet data is used for the first stage training of a two-stage hybrid expert-conditional LoRA fine-tuning strategy to improve the model's blind target localization ability and form priors for spatial localization and synthesis. Consistency triplet data is used for the second stage training to improve the consistency of the model's accessory texture, ensuring high consistency between the replaced accessory and the target reference. Finally, the text prompts and image data are integrated to obtain the input data for the first stage, represented as follows: The input data for the second stage is represented as The dataset was divided into training, validation, and test sets in a ratio of 8:1:

1.

4. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The method for LoRA training using the two-stage hybrid expert-conditional LoRA fine-tuning strategy in step S5 is as follows: The training process is divided into two phases. In the first phase, a spatial localization expert and other LoRA experts learn how to redraw accessory points on unpaired images and perform natural fusion. This phase is a weakly supervised training process mainly used to improve the model's spatial localization ability. In the second phase, other expert LoRA models are trained to maintain consistency in texture and shape between the accessory mask region of the clothing model image and the input accessory image. At the same time, both phases of training introduce the Conditional Multimodal DiT Attention method (CMMDiTA) and the Attention Conditional Position Masking mechanism (ACPM) to conditionally perceive and restrict attention, enabling the model to obtain multi-conditional scalability and precise local control.

5. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The conditional multimodal DiT attention method in step S5 is as follows: For feature vectors derived from model images, text prompts, accessory images, and mask images: in, It is the first The conditional encoding vector of an accessory is composed of a weighted sum of the feature vector of the accessory image and the positional features of the mask image. First, the input feature sequence... The query matrix is ​​divided into parts derived from the main clothing model images and text. The matrix is ​​defined as a token that allows global access, ensuring overall consistency. Correspondingly, the query matrix will be based on the accessory conditions. Defined as only being able to access its corresponding local subsequence features The token is used to avoid interference between different accessory conditions. Then, multimodal attention is applied. For the global branch, the text, image subject and all conditional information features are fused. For the conditional branch, attention is calculated separately for each accessory condition to ensure that feature alignment is performed only in a local range. Finally, the results from the global branch and the conditional branch are concatenated and then fed into the next module through a linear projection layer.

6. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The method for two-stage model optimization using the joint loss function in step S6 is as follows: The model training process is divided into two stages. In the first stage, a localization expert (LoRA) and other LoRA experts are trained to enable the model to learn how to regenerate accessory elements and their natural integration with boundaries at key accessory locations in unpaired images. This is mainly achieved through joint diffusion denoising loss. Background retention loss With expert load balancing loss The model is optimized; in the second stage, other hybrid expert LoRAs are trained to ensure high consistency with the target accessory, mainly through joint diffusion denoising loss. Consistency loss Background retention loss Pixel-perceptual loss and expert load balancing loss Optimize the model to ensure stable learning of conditional image information.

7. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The method for fusing the model weights and network structure of the multi-condition hybrid expert LoRA and DiT trained in step S7 is as follows: First, the pre-trained DiT backbone model weights are loaded, including the original weights of all Transformer layers. At the same time, the trained MOE-C-LoRA weights are loaded, which store the corresponding low-rank decomposition matrix for each conditional expert LoRA. Then, a dynamic conditional fusion strategy is adopted. During inference, dynamic gating coefficients are calculated based on the input conditional features, and then the router dynamically selects and weights the expert LoRAs. Subsequently, the fused weights are replaced in the corresponding layers of the DiT model, and the fused overall weights are saved as new model checkpoints for inference. Finally, the model network results and the model weights with fused LoRA expert parameters are loaded. With the help of the CMMDiTA and ACPM modules, the model uses the weights to calculate attention and feature updates, gradually spreading the results.

8. The method for partial replacement of clothing accessories according to claim 1, characterized in that, The method for inference using the fused DiT based on the user-input clothing model image, accessory image, custom mask image, and text description in step S7 is as follows: S71: First, the system requires users to upload images of clothing models and accessories. Then, the system applies a custom mask to the clothing model images to determine the area on the upper body of the accessories and obtains the mask image. At the same time, the system performs preprocessing operations on the input clothing model images, accessory images, and mask images to ensure that the image data is suitable for subsequent model calculations. In addition, the relevant text descriptions entered by the user will also be transferred to the system for later use. S72: The preprocessed image and text description are combined and input in parallel into the Flux model with weight correction applied by MOE-C-LoRA. The CMMDiTA masking mechanism is used in the attention layer. First, a VAE encoder maps the clothing model image to latent space noise. Then, a CLIP visual encoder is used to extract the image feature vector of the accessory image. Subsequently, a position encoder is used to encode the mask image into a position vector. Next, a T5 text encoder converts the text description into a semantic feature vector. Finally, the image feature vector and the position encoded vector are weighted and added together, which makes it easier to effectively control the noise influence range and accessory appearance features in subsequent attention calculations. S73: Input the features obtained in S72 above into the DiT architecture. The system will start from pure noise and gradually estimate the latent space representation of the denoised version of the clothing model image in the latent space. The features are continuously fine-tuned by applying different expert LoRA. In addition, the Conditional Multimodal DiT Attention Method (CMMDiTA) is used to ensure the stability of the feature space position, and the Attention Conditional Position Masking Mechanism (ACPM) is used to ensure the consistency of the accessory features. S74: Input the latent space representation of the clothing model image obtained in S73 into the pre-trained VAE decoder module for decoding and reconstruction to obtain the final output clothing model image. This model image contains a high-quality image with clear edge blending, texture structure and consistent appearance after the target accessory is replaced in the specified location area, thus realizing the user-specified target accessory replacement task. S75: The system outputs the corresponding user-input-generated clothing model image with target accessories to the user interface for real-time preview, and can export and save it as JPG, PNG or PDF formats, improving the image's secondary editing capabilities and application flexibility.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for partial replacement of clothing accessories as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; A storage system for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the partial replacement method for clothing accessories as described in any one of claims 1 to 8.

Images