Knowledge enhancement generation and mask constraint local modification method for clothing customization text-to-image agent system

Abstract

The knowledge enhancement generation and mask constraint local modification method of the clothing customization text-to-image agent system combines the clothing knowledge graph, the retrieval enhancement generation RAG process and the mask constraint local modification, converts the clothing category attributes, manufacturer quality rule prompts and user feedback into generation prompt constraints and modification control conditions, and realizes the knowledge enhancement generation and controllable local modification of the clothing image without fine-tuning or retraining the basic text-to-image generation model. The retrieval enhancement generation (RAG) process driven by the clothing knowledge graph and the mask constraint local redrawing solve the problems of the general model in the professional term understanding, the stability of local modification and the controllability of user interaction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence image generation technology and clothing digitization, specifically to a knowledge enhancement generation method and a mask constraint local modification method based on clothing knowledge graph and retrieval enhancement generation RAG process, applicable to vertical application scenarios such as clothing customization, clothing design, e-commerce clothing display, and virtual try-on. Background Technology

[0002] With the rapid development of diffusion model technology, large-scale text-based graph models, such as Stable Diffusion, DALL-E 3, and Flux, have been widely applied in creative design, advertising production, and game art. Meanwhile, the continuous improvement in inference efficiency and generation quality of diffusion models has created conditions for refined applications in vertical industries. In the apparel industry, users can input their clothing requirements via natural language, and the text-based graph agent can quickly generate clothing renderings to assist design decisions, shorten design cycles, and reduce manual design costs.

[0003] However, the general-purpose text image model faces the following technical limitations in its application in the apparel vertical industry: First, there are limitations in the understanding of clothing categories and attributes. The training data for general-purpose models comes from open images on the internet, lacking a systematic understanding of professional clothing terminology. The clothing industry has a highly specialized terminology system. For example, neckline types include more than ten types such as stand-up collars, turn-down collars, suit collars, off-the-shoulder collars, and V-necks; cuff types include various forms such as cuff sleeves, ribbed sleeves, batwing sleeves, and bell sleeves; and fabric names involve a large number of professional terms such as flannel, Oxford cloth, chiffon, and georgette. General-purpose models struggle to accurately distinguish between subcategories such as Hanfu ruqun (traditional Han Chinese clothing), work jackets, and baseball jackets. The generated clothing images often deviate from user descriptions in key attributes, forcing designers to repeatedly modify prompts and experiment, severely reducing work efficiency.

[0004] Second, the stability of local modifications is insufficient. When users request local modifications to the generated results, existing models typically use a global redraw, causing unmodified attributes to drift. For example, if a user requests to "change the white shirt to black," the model might generate a black skirt, a black T-shirt, or change the collar and cuff styles of the shirt. In clothing customization scenarios, users often need to fine-tune individual attributes while maintaining the overall design style. The global redraw approach cannot meet such needs for controllable local modifications, and each redraw requires the user to re-evaluate all attributes, resulting in extremely high interaction costs.

[0005] Third, the modification process lacks a user intervention mechanism. Existing text-based image modification systems typically follow a single-loop process: "user input command → model execution → output result." When the automatically modified result does not meet user expectations, the user can only initiate a new round of modifications by re-entering the command, lacking fine-grained control over the modification process. Especially in scenarios involving multiple attribute modifications, even if the automatic model verification mechanism determines the modification is "successful," the actual result may still deviate from the user's subjective aesthetic judgment. Therefore, a user confirmation step is needed to improve system usability.

[0006] Fourth, the engineering cost of introducing domain knowledge is high. Traditional domain adaptation solutions mainly rely on full parameter fine-tuning, which requires large-scale manually labeled clothing image data and is difficult to quickly extend to emerging clothing categories. Although parameter-efficient fine-tuning techniques (such as LoRA) can reduce training costs, they still require maintaining independent model weight files, which poses a high maintenance burden in production environments with frequent model version iterations.

[0007] In recent years, Retrieval-Enhanced Generative Algorithm (RAG) technology has provided a low-cost and highly flexible new path for injecting domain knowledge into large models. This technology dynamically retrieves external knowledge bases during the inference phase and injects the retrieved domain knowledge into the model input, thereby enhancing the model's domain understanding capabilities without modifying the weights of the basic text-based image generation model. Applying the RAG concept to the text-based image domain, combined with a clothing knowledge graph, is expected to significantly improve the attribute accuracy of clothing image generation without fine-tuning or retraining the basic text-based image generation model, and support incremental expansion of few-sample / low-sample knowledge configuration for new product categories. Furthermore, visual-linguistic pre-trained models in the clothing domain, such as FashionCLIP, provide a unified feature representation foundation for semantic retrieval and attribute recognition of clothing knowledge, serving as a key technological support for constructing the aforementioned system. Summary of the Invention

[0008] This invention aims to overcome the aforementioned shortcomings of existing technologies by providing a knowledge enhancement generation and mask constraint local modification method for a clothing customization text graph agent system. Through a retrieval enhancement generation RAG process driven by a clothing knowledge graph and local redrawing of mask constraints, it addresses the deficiencies of general models in terms of professional terminology understanding, stability of local modifications, and controllability of user interaction.

[0009] The objective of this invention is achieved through the following technical solution: a method for knowledge enhancement generation and local modification of mask constraints in a clothing customization text-based image agent system, comprising the following steps: (1) Retrieval Enhancement Generation Based on Clothing Knowledge Graph: Construct a clothing knowledge graph containing clothing categories, clothing attributes, terminology mapping and manufacturer quality rule prompts, and establish a semantic retrieval index for the clothing knowledge graph; after receiving the user's natural language clothing generation request, execute the retrieval enhancement generation RAG process based on the semantic retrieval index and the clothing knowledge graph. The RAG process includes candidate clothing knowledge recall, constraint reasoning, candidate node reordering and prompt word enhancement; input the obtained standard attribute descriptions, manufacturer quality rule prompts and reference examples into the basic text-based image generation model to generate initial clothing images; (2) Mask-constrained local modification: Receive the user's natural language modification instruction for the initial clothing image, parse it to obtain the modification object, target attribute and retained attribute; generate a clothing instance-level or part-level spatial mask according to the modification object, and perform mask-constrained local redrawing of the initial clothing image based on the spatial mask; perform attribute consistency verification on the redrawing result, and generate a confirmation item based on the manufacturer's quality rule prompt, and display the attribute verification report, manufacturer's quality rule prompt and confirmation item to the user; output the final clothing image after user confirmation.

[0010] Furthermore, the construction of the clothing knowledge graph in step (1) includes the following sub-steps: (1.1) Hierarchical structure design: A three-level category hierarchy is established. The first level is the clothing category, including tops, bottoms, coats, dresses, and accessories; the second level is the clothing sub-categories, including shirts, T-shirts, suits, and jeans; the third level is the sub-categories, including formal shirts, casual shirts, and flannel shirts. General attributes and exclusive attributes are associated with each sub-category. General attributes include color, fabric, length, and pattern, while exclusive attributes include collar type, cuff type, and placket type. At the same time, a manufacturer-specific quality constraint sub-graph is established, which represents the manufacturer, manufacturer quality standard documents, appearance quality rules, sewing quality rules, size deviation rules, defect types, inspection items, and acceptance thresholds as graph nodes, and connects them with clothing category, attribute, and part nodes through relationships such as "define", "include", "constraint", and "applicable". (1.2) Terminology mapping and feature storage: For each node, the mapping relationship between standard terms, industry aliases, and user colloquialisms is stored; for clothing category, attribute, part, and defect type nodes with visual appearance features, the FashionCLIP model is called to encode the standard terminology text description and several standard example images respectively, and the average value of the resulting embedding vectors is taken as the visual-semantic fusion feature vector of the node; for rule-based nodes such as manufacturers, manufacturer quality standard documents, inspection items, allowable deviations, and acceptance thresholds, text encoding or rule vector encoding methods are used to store them in the vector database; (1.3) Bidirectional Index Construction: A bidirectional mapping relationship is established between the nodes of the clothing knowledge graph and the original text documents. The original text documents include general clothing terminology standards, enterprise garment quality standards documents, and appearance quality standards documents. The quality requirements, defect levels, appearance acceptance items, sewing requirements, allowable size deviations, and color difference requirements in the documents are structurally extracted and mapped to the corresponding clothing categories, parts, and attribute nodes. At the same time, the attribute legality constraint table and the manufacturer quality rule constraint table are maintained to record the legal value combinations of each attribute under each category and the quality acceptance conditions corresponding to specific manufacturers. These are used for constraint reasoning, candidate node clipping, prompt word construction, and manufacturer quality rule prompts in the RAG process. For complex quality items such as sewing quality, size deviation, thread ends, stains, and color difference levels that are difficult to automatically determine based solely on the stability of the generated image, the system converts them into generation prompt words, negative prompt words, and manufacturer quality rule prompts and confirmation items. If necessary, they enter the manual review or pending review process.

[0011] Furthermore, the RAG process in step (1) includes the following sub-steps: (1.4) User Intent Parsing and Semantic Retrieval: Receive user natural language input and extract clothing category keywords, attribute keywords, and manufacturer or manufacturer quality standard keywords; when the input contains a clear manufacturer, category, or manufacturer quality standard name, first limit the candidate category, attribute, and manufacturer quality rule range based on the clothing knowledge graph, then call the FashionCLIP text encoder to embed the user input and perform approximate nearest neighbor retrieval within the limited range; when the input is relatively ambiguous, first recall the Top-K semantically similar nodes in the vector database, then perform constraint reasoning and reordering based on the hierarchical structure of the clothing knowledge graph on the candidate nodes, and prioritize returning sub-category nodes that are consistent with the user intent in terms of category level, attribute set, manufacturer quality rules, and legal attribute combination; (1.5) Construction of enhanced search prompts: The retrieved standard terms, attribute constraint list, legal attribute combinations, manufacturer quality rule prompts, defect avoidance requirements and example image paths are assembled into a structured context; the enhanced prompts are constructed according to the template "[standard category name] + [attribute constraint list] + [manufacturer quality rule prompts] + [defect avoidance requirements] + [style reference description], solid color background, front view"; and complex quality items are output as negative prompts or manufacturer quality rule prompts and confirmation items; the example images are injected into the diffusion image generation model that supports image reference input in the form of image reference or IP-Adapter, guiding the generation results to align with the standard visual features and manufacturer quality rules in the clothing knowledge graph; (1.6) Incremental knowledge adaptation: When adding a new category, a new manufacturer, or a new manufacturer's quality standard document, the corresponding node information, attribute constraints, manufacturer quality rules, and standard example diagrams are entered into the clothing knowledge graph; FashionCLIP or a text encoder is called to encode the new node and its rule description and update the vector index. Without fine-tuning or retraining the basic text graph generation model, it can support incremental expansion of the knowledge configuration of new categories, new manufacturers, and manufacturer quality rules with few or low samples.

[0012] Furthermore, the local modification of the mask constraint in step (2) includes the following sub-steps: (2.1) Structured instruction parsing: Based on the clothing knowledge graph, the user's natural language modification instructions are parsed into a standardized JSON format, including the object to be modified, the list of modified attributes, and the list of retained attributes; the existence of the attributes in the parsing result is verified in the clothing knowledge graph; (2.2) Precise region mask generation: Call the clothing instance segmentation model to segment the region corresponding to the modified object from the initial clothing image and generate a pixel-level binary mask; when the modified object corresponds to local parts such as collar, cuff, placket, pocket, and hem, the part category output based on the instance segmentation model or combined with general segmentation models such as SAM is used to refine the mask into a secondary mask, and the clothing part level mask is then processed by morphological dilation. (2.3) Mask-constrained local redrawing: The style embedding vector and lighting parameters of the initial clothing image are extracted as constraints. Only the area within the mask is redrawn with diffusion. The number of redrawing steps is set to 20-30 steps, and the automatic initial redrawing intensity is set to 0.5-0.7. A progressive redrawing strategy is adopted. The first 50% of the steps are used to modify the area within the mask, and the last 50% of the steps are used for edge blending and detail refinement. If automatic retry is triggered, the redrawing intensity increases by 0.05 each time, and the automatic retry limit does not exceed 0.85. (2.4) Attribute-level consistency verification: The trained or fine-tuned FashionCLIP-ResNet50 attribute recognition model is called to predict the attributes that can be automatically recognized in the initial clothing image and the redrawn clothing image respectively. The attributes that can be automatically recognized include category, color, collar type, cuff type, pattern, fabric; length, placket type and other attributes not included in the prediction range of the attribute recognition model are treated as constraints of the clothing knowledge graph, local redraw prompts or user confirmation items. Among them, the content involving the manufacturer's quality rules is treated as manufacturer's quality rule prompts and confirmation items. Verify whether the retained attributes are consistent and whether the modified attributes are successfully changed to the target value. If the verification fails, adjust the redraw intensity (increase by 0.05 each time and not exceed 0.85) and redraw. The maximum number of automatic retries is 3. (2.5) User Interaction Confirmation: When the attribute consistency check passes, or fails but reaches the preset maximum number of automatic retries, the system pushes the current optimal redraw result along with the attribute check report, manufacturer quality rule prompts, and confirmation items to the user confirmation interface. The attribute check report visually marks the check status (pass / fail) of the attributes that support automatic identification and compares the target value with the measured value. The manufacturer quality rule prompts and confirmation items list prompts for appearance defects, sewing quality, size deviation, color difference, stains, loose threads, or other manufacturer quality rule constraints, and marks items that cannot be reliably and automatically determined as user confirmation items. The system allows users to either accept the current result or undergo manual review. Users can choose one of the following actions: accept the current result and output the final garment image; manually adjust the redrawing intensity, inference steps, local redrawing prompts, or negative prompts to trigger a new round of redrawing; or abandon the current modification and retain the initial garment image. The system will write the user's acceptance, rejection, prompt correction, parameter adjustment, and quality issue feedback into the user preference sample library, the generated effect sample library, and the manufacturer's quality rule pending review library, respectively. Feedback involving changes to the manufacturer's quality standard document or the addition of defect rules will only enter the pending review process and will be written into the manufacturer's personalized quality constraint sub-graph after manual review and confirmation.

[0013] The technical solution of this invention can be summarized as follows: 1. A knowledge enhancement generation method based on a clothing knowledge graph and a retrieval-enhanced generation RAG process is proposed. By constructing a hierarchical clothing knowledge graph and a FashionCLIP vector index, clothing professional knowledge is dynamically retrieved and injected into the generation prompts during the inference stage. This improves the recognition accuracy of sub-categories and attributes without modifying the weights of the basic text-based graph generation model. When adding new categories, new manufacturers, or new manufacturer quality standard documents, only the clothing knowledge graph and vector index need to be updated, demonstrating efficient incremental expansion capabilities for few / low-sample knowledge configuration. 2. A method for local modification of mask constraints is proposed. Spatial masks are generated by clothing instance segmentation and part-level mask refinement. Style preservation constraints are combined to perform local redrawing of mask constraints. The FashionCLIP-ResNet50 attribute recognition model is used to perform consistency verification on attributes that support automatic recognition, effectively suppressing unexpected changes to unmodified attributes. 3. A human-machine collaborative modification confirmation mechanism is proposed. After the automatic verification process is completed, a user interaction confirmation step is introduced. Through a visual attribute verification report and an adjustable parameter interface, users are included in the modification quality control closed loop, improving the system's usability and controllability in complex modification scenarios.

[0014] The beneficial effects of this invention are: Firstly, by using the retrieval-enhanced generation (RAG) technology, professional knowledge from the clothing knowledge graph is dynamically injected into the generation prompts, which enhances the basic text-based image generation model's ability to understand subdivided clothing categories and professional terms, while avoiding the training costs and maintenance burdens caused by fine-tuning or retraining the basic text-based image generation model. Secondly, by using the mask constraint local redrawing mechanism and FashionCLIP-ResNet50 attribute consistency verification, the attributes that support automatic recognition are automatically verified, and the manufacturer quality rule prompts and confirmation items are provided for complex manufacturer quality rules, which effectively suppresses the drift phenomenon of unmodified attributes during the modification process. Third, by introducing a user interaction confirmation process, human judgment is incorporated into the quality control process, thus bridging the gap between automatically verified indicators and user subjective satisfaction. Fourth, incremental knowledge adaptation only requires updating the nodes and vector indices of the clothing knowledge graph, without needing to fine-tune or retrain the basic text-to-graph generation model. It supports the rapid deployment of new categories, new manufacturers, and manufacturer quality rules with few or low-sample knowledge configuration.

[0015] This invention has the advantages of high recognition accuracy, high modification precision, high portability, and low deployment cost, and can provide a complete and feasible technical path for the engineering application of text-based image technology in the clothing customization industry. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall system architecture of the present invention, showing the composition and interaction relationship of the clothing knowledge graph, the retrieval enhancement and generation RAG module, and the mask constraint local modification module; Figure 2 This is a flowchart of the retrieval enhancement generation process based on the clothing knowledge graph of the present invention, which shows the complete process of clothing knowledge graph construction, manufacturer quality rule prompts and constraints, FashionCLIP vector index construction, semantic retrieval, graph constraint reasoning and retrieval enhancement prompt word construction and incremental knowledge adaptation; Figure 3 This is a flowchart of the local modification of mask constraints according to the present invention, which shows the complete closed-loop process of instruction parsing, part-level mask generation, local redrawing of mask constraints, FashionCLIP-ResNet50 attribute-level verification, manufacturer quality rule prompts and confirmation items / manual review, and user feedback write-back. Detailed Implementation

[0017] The core technology of this invention is based on a clothing knowledge graph, integrating a retrieval-enhanced RAG generation module with mask constraint local modification technology to solve the limitations of general text-based large-scale models in category recognition and the instability of local modifications in the field of clothing customization. Specifically, the RAG module executes the retrieval-enhanced RAG generation process, and the mask constraint local modification module executes the mask constraint local modifications.

[0018] The present invention proposes a knowledge-enhanced generation and mask constraint local modification method for a clothing customization text-based image agent system, such as... Figure 1 and Figure 2 This includes the following steps: (1) Enhanced retrieval generation based on clothing knowledge graph A hierarchical clothing knowledge graph is constructed to ensure the standardization and completeness of clothing knowledge from the underlying data structure. At the same time, enterprise garment quality standard documents, appearance quality standard documents, and user interaction feedback are integrated into the clothing knowledge graph as personalized knowledge sources, so that the same clothing category can form differentiated constraints based on different manufacturers' quality rules, manufacturer quality rule prompts, and user preferences. Based on the FashionCLIP model, visual-semantic vector indexes are constructed for nodes with visual appearance features, and text or rule vector indexes are constructed for rule-type nodes such as manufacturer quality standard documents, inspection items, and acceptance thresholds.

[0019] (1.1) Construction of a hierarchical clothing knowledge graph: Using the Neo4j graph database, a three-level category hierarchy was established based on the national standard GB / T15557-2008 Clothing Terminology. The first level is the clothing category, including five major categories: tops, bottoms, outerwear, dresses, and accessories; the second level is the clothing subcategories, including 32 subcategories such as shirts, T-shirts, suits, and jeans; the third level is the detailed categories, including 127 detailed categories such as formal shirts, casual shirts, and flannel shirts. In addition to the general category graph, a manufacturer-specific quality constraint subgraph was further established. The enterprise's garment quality standard document and appearance quality standard document were parsed into nodes such as Manufacturer, QualityStandardDocument, QualityRule, DefectType, InspectionItem, and Tolerance, and connected to clothing category, garment part, attribute, and defect type nodes through relationships such as AppliesTo, Constraints, Contains, and Adopts. For example, requirements such as appearance quality, sewing quality, color difference, stains, loose threads, and size deviation in a manufacturer's garment quality standard document are mapped to corresponding quality rule nodes, so that the same "shirt" node has different generation constraints and acceptance conditions under different manufacturers' quality rules.

[0020] (1.2) Terminology Mapping and Vector Index Construction: For each sub-category, general and specific attributes are associated, and the mapping relationship between standard terms, industry aliases, and user colloquialisms is stored. For categories, attributes, parts, and defect type nodes with visual appearance features, FashionCLIP (based on CLIP-ViT-B / 32 backbone, pre-trained on Fashion-200K and DeepFashion datasets) is called to encode the standard terminology text and several standard example images respectively, and the mean of the embedding vectors is stored in the FAISS vector database; for rule-based nodes such as manufacturers, manufacturer quality standard documents, inspection items, allowable deviations, and acceptance thresholds, text encoding or rule vector encoding is used to store them in the FAISS vector database, and a cosine similarity retrieval index is constructed.

[0021] (1.3) Bidirectional Index Construction and Attribute Legality Constraints: A bidirectional mapping relationship is established between the nodes of the clothing knowledge graph and the original text documents, supporting semantic-based node retrieval and structure-based attribute query. The original text documents include national clothing terminology standards, enterprise garment quality standards documents, and appearance quality standards documents. The system performs paragraph-level and table-level parsing on the manufacturer's quality standard documents, extracting fields such as "applicable category, testing items, defect type, quality grade, allowable deviation, acceptance threshold, and handling opinions," and mapping them to the corresponding category nodes, attribute nodes, and manufacturer quality rule nodes. At the same time, the system maintains the attribute legality constraint table and the manufacturer quality rule constraint table, recording the legal value combinations of each attribute under each category, as well as the appearance quality, sewing quality, size deviation, and defect tolerance requirements under specific manufacturers. The aforementioned constraint table is stored in Neo4j node attributes in JSON format. During the retrieval phase, it is used for graph constraint reasoning and candidate node pruning. During the local modification phase, it is used to generate manufacturer quality rule prompts, negative prompt words, and manufacturer quality rule prompts and confirmation items. Among them, complex quality items such as sewing quality, size deviation, thread ends, stains, and color difference level require user confirmation or manual review and are not directly equivalent to fully automatic quality inspection.

[0022] (1.4) User Intent Parsing and Semantic Retrieval (Retrieval Enhancement Generation RAG Process): Receives user natural language input (e.g., "a stand-up collar flannel formal shirt generated according to Macondo garment quality standards"), and extracts keywords related to category, attribute, scenario, manufacturer, or manufacturer quality standard. The system adopts a hybrid retrieval process combining graph constraints and vector retrieval: When the input contains a clear manufacturer, category, or manufacturer quality standard name, it first locates the corresponding manufacturer's personalized quality constraint subgraph and candidate category range along the clothing knowledge graph, then calls the FashionCLIP text encoder to encode the input, and retrieves Top-K similar nodes from the FAISS index within the limited range; when the input is relatively ambiguous, it first performs global vector retrieval to recall Top-K candidate nodes, and then performs constraint reasoning, candidate node pruning, and reordering based on the three-level category hierarchy of the clothing knowledge graph, legal attribute combinations, and manufacturer quality rules. Specifically, if a candidate node is located at the third-level sub-category level, the attribute constraints, manufacturer quality rules, and example images bound to that node are directly read. If the candidate node is located at the second-level sub-category level, all third-level sub-categories are expanded downwards along the HasChild relationship, and then reordered based on attribute keywords, manufacturer quality rule matching degree, and semantic similarity. If the candidate node is located at the first-level major category level, it is first narrowed down to the second-level sub-category based on the scenario and attributes in the user input, and then further expanded to the third-level sub-categories. The comprehensive score of the candidate node is a weighted sum of semantic similarity, hierarchical matching degree, attribute consistency score, and manufacturer quality rule matching degree, and the sub-category node with the highest score is output first.

[0023] (1.5) Construction of Enhanced Search Tips: The retrieved standard terms, attribute constraint lists, legal attribute combinations, manufacturer quality rule tips, defect avoidance requirements, and example images are assembled into enhanced search tips according to a template. The format is: "[Standard Category Name], [Attribute Constraint List], [Manufacturer Quality Rule Tips], [Defect Avoidance Requirements], [Fabric Texture Description], Solid Color Background, Front View, Professional Apparel Photography Style". Among them, the manufacturer quality rule tips are used to constrain the results to conform as much as possible to the specific manufacturer's requirements in terms of appearance quality, sewing quality, size deviation, color difference, stains, loose threads, and defect level during the generation stage. For complex quality items that are difficult to be stably and automatically determined by the image model, the system adds them to the model input and confirmation interface in the form of negative tips, manufacturer quality rule tips, or manufacturer quality rule tips and confirmation items. The retrieved standard example images are injected into diffusion-type image generation models such as Flux.1 or Stable Diffusion XL, which support image reference input, local redrawing, or mask condition control, through the IP-Adapter method to guide the visual style of the generated results to align with the standard features of the apparel knowledge graph and the manufacturer quality rules. Call the basic text image generation model to perform inference and output the initial clothing image.

[0024] (1.6) Incremental Knowledge Adaptation: When adding new categories such as "New Chinese Hanfu", new manufacturers, or new manufacturer quality standard documents, the administrator adds corresponding nodes in Neo4j and enters standard terms, aliases, attribute constraints, manufacturer quality rules, and standard example images. For newly added manufacturer quality standard documents, the system parses the standard documents into manufacturer quality rule nodes and establishes relationships with applicable categories, parts, and attributes; for newly added user-common expressions or manually corrected expressions, the system writes them into the terminology mapping relationship. The system calls FashionCLIP or a text encoder to encode the new nodes, rule descriptions, and example images, and updates the FAISS index. Throughout the process, there is no need to fine-tune or retrain the basic text-to-image generation model. New categories, new manufacturers, or newly added user expressions can take effect in the next retrieval inference through a few-sample / low-sample knowledge configuration method.

[0025] (2) Local modification of mask constraints

[0026] like Figure 3 The system parses the user's natural language modification instructions into a structured format, generates precise region masks by segmenting clothing instances, performs local redrawing only on the regions within the mask, and ensures that the modification results meet the requirements through attribute-level consistency checks and user interaction confirmation.

[0027] (2.1) Structured Modification Command Parsing: Based on Qwen2-7B-Instruct or other large language models, a fixed Prompt template is written to parse the user's natural language modification commands into a standardized JSON format. For example, if the user enters "Change this shirt to black", the parsed result is: {"Modified Object":"Shirt","Modified Attribute":[{"Name":"Color","Original Value":"White","Target Value":"Black"}],"Retained Attribute":["Category","Neckline","Cuff","Size","Fabric"]}. The system automatically verifies whether the attributes in the parsed result exist in the clothing knowledge graph; if not, the user is prompted to re-enter.

[0028] (2.2) Precise Region Mask Generation: The trained or fine-tuned YOLOv8-Seg model is used to perform instance segmentation on the initial clothing image, segmenting clothing instance regions such as shirts, and generating a binary mask M. If the user modifies the object to clothing parts such as collar, cuffs, placket, pockets, or hem, a part-level segmentation head or a general segmentation model such as SAM is used to refine the mask based on the clothing instance mask, resulting in a part-level mask M_part. OpenCV is used to perform morphological dilation of 3-5 pixels on the mask to obtain M_dilate, in order to avoid obvious splicing marks at the edges of the redrawn areas.

[0029] (2.3) Mask-Constrained Local Repainting: A diffusion-based image generation model supporting local repainting or mask condition control, such as Flux.1-Fill or Stable Diffusion Inpainting, is invoked, loading the initial clothing image and the generated mask. The style embedding vector of the initial clothing image (extracted via CLIP or VGG features) and the average brightness, contrast, and color temperature of the modified area are extracted as constraints. Automatic repainting parameters are set as follows: initial denoising strength = 0.6, inference steps = 25, guidance scale = 7.5; if automatic retry is triggered, the repainting strength is increased by 0.05 each time, with an upper limit not exceeding 0.85. A progressive repainting strategy is adopted, modifying the masked area only in the first 50% of the diffusion process (e.g., the first 12 steps), and using the remaining 50% for edge blending and detail refinement.

[0030] (2.4) Attribute-level consistency verification: The trained or fine-tuned FashionCLIP-ResNet50 attribute recognition model (an MLP classification head is added to the FashionCLIP visual encoder, and DeepFashion2 attribute annotation data can be used for supervised fine-tuning, supporting automatic attribute prediction for categories, colors, collar types, cuff types, patterns, fabrics, etc.) is invoked to perform predictions for automatically recognized attributes on the initial and redrawn clothing images respectively. Length, placket type, and other attributes not included in the model's prediction scope are handled by clothing knowledge graph constraints, local redraw prompts, or user confirmation items; among them, content involving manufacturer quality rules is handled as manufacturer quality rule prompts and confirmation items.

[0031] Verification logic: - Preserve attribute predictions that are completely consistent with the initial garment image → By - Modify the attribute prediction value to equal the target value → Pass - If any of the above conditions are not met → the verification fails. Adjust the redraw intensity (increase by 0.05 each time and the automatic retry limit does not exceed 0.85) and then re-execute step (2.3). The maximum number of automatic retryes is 3.

[0032] (2.5) User Interaction Confirmation: When the attribute consistency check passes, or fails but reaches the preset maximum number of automatic retries, the system pushes the current optimal redraw result, attribute check report, manufacturer quality rule prompts and confirmation items to the user confirmation interface. The attribute check report is presented in a four-column table (attribute name / original value / target value / measured value), and items that fail in the attributes that can be automatically identified are highlighted; the manufacturer quality rule prompts and confirmation items display prompts corresponding to appearance quality, sewing quality, size deviation, color difference, stains, thread ends and other defect avoidance requirements, among which complex quality items that are difficult to be reliably judged automatically are marked for user confirmation or manual review.

[0033] Users can choose the following operations: - Accept current result: Use the current redrawn clothing image as the final output; - Manually adjust parameters and retry: Users can adjust the redraw intensity (0.5~0.9, step size 0.05), inference steps (20~50 steps), local redraw prompt words and negative prompt words, and the system will re-execute steps (2.3) to (2.5). - Discard this modification: Keep the initial clothing image and terminate this modification process.

[0034] User operation records and parameter adjustments are written to the session log and associated with the apparel knowledge graph nodes, manufacturer quality rule prompts, generated prompt words, masked areas, and output images found in this search. This information is used for subsequent parameter optimization and incremental updates of the apparel knowledge graph. For results accepted by the user, the system writes the corresponding image, prompt words, and attribute combinations as positive samples into the user preference sample library and the generated effect sample library, and updates the vector index of the relevant visual nodes. For results rejected by the user or manually corrected, the system extracts failure attributes, corrected expressions, and quality problem labels to update the terminology mapping table, attribute weights, and candidate re-ranking parameters. For feedback involving changes to manufacturer quality standard documents or the addition of defect rules, the system only generates manufacturer quality rule items pending review. These items are written into the manufacturer's personalized quality constraint subgraph only after manual review and confirmation, avoiding the direct writing of user subjective preferences into the manufacturer quality rule nodes.

[0035] Implementation Examples

[0036] Example 1: Garment generation and partial modification based on Macondo garment quality standards and woven garment appearance quality standards

[0037] This embodiment uses the "Macondo (Beijing) Culture Co., Ltd. Garment Quality Inspection Standard" (document number MCFC-001-2023) and "Q-SM 1323-2022 Appearance Quality Standard (Woven Garments)" as the manufacturer's quality standard documents to illustrate the process of generating and partially modifying a woven shirt.

[0038] The user inputs a natural language request: "Generate a dark gray woven stand-up collar long-sleeved shirt with a slim fit, straight placket, and front view, according to the Macondo ready-to-wear quality standard and the Q-SM 1323-2022 woven garment appearance quality standard." The system parses and obtains the manufacturer's quality standard keywords "Macondo ready-to-wear quality standard" and "Q-SM 1323-2022 woven garment appearance quality standard," the category keyword "woven shirt," and the attribute keywords "dark gray, stand-up collar, long sleeves, slim fit, straight placket, and front view."

[0039] The system first locates the Manufacturer node, QualityStandardDocument node, and the "Woven Shirt" subcategory node along the clothing knowledge graph, and reads the general and specific attributes bound to this subcategory node, including color, fabric, pattern, length, collar type, cuff type, and placket type. Then, the system reads the manufacturer quality rule prompts corresponding to the two manufacturer quality standard documents mentioned above, including clean appearance, no loose threads or lint, no dirt or holes, no obvious color difference, symmetrical left and right sides, straight and flat placket, smooth collar shape, no obvious broken or skipped stitches, and buttons, zippers, and other accessories are free of defects and consistent with the approved sample.

[0040] The system constructs enhanced prompts: "Woven stand-up collar long-sleeved shirt, dark gray, slim fit, straight and smooth placket, rounded collar, front view, solid color background, professional clothing photography style, meets Macondo garment quality inspection standards and Q-SM1323-2022 woven garment appearance quality requirements, clean appearance, symmetrical left and right, complete accessories." Simultaneously, it constructs negative prompts: "Loops, lint, stains, holes, obvious color difference, left and right asymmetry, skewed placket, skipped stitches, broken threads, seams, ripped edges, uneven ironing, zipper arching, missing accessories." The system inputs the retrieved standard example images into a diffusion-based image generation model that supports image references via an IP-Adapter or image reference method, and outputs the initial garment image.

[0041] After the initial image is generated, the user inputs a local modification command: "Change this shirt to black, keep other attributes unchanged." The system parses this modification command into a structured format: the object to be modified is "shirt," the attribute to be modified is "color: change dark gray to black," and the attributes to be retained include "category, collar type, cuff type, placket type, pattern, fabric, and front view style." The system calls a trained or fine-tuned YOLOv8-Seg model to generate a shirt instance mask; if the user further requests "only adjust the placket color or placket shape," then a placket part-level mask is generated based on the shirt instance mask, combined with a part-level segmentation model or SAM.

[0042] The system invokes a diffusion-based image generation model that supports local redrawing or mask-conditional control to perform mask-constrained local redrawing. The initial redrawing intensity is set to 0.6, the inference steps are set to 25, and the guiding scale is set to 7.5. During the redrawing process, the system uses "black woven shirt, keep the stand-up collar, cuffs, and placket straight, and maintain the slim fit" as local redrawing prompts, and "change the collar type, change the placket, produce obvious loose threads, stains, holes, color differences, left and right asymmetry, skipped stitches, and broken threads" as negative prompts.

[0043] After redrawing, the system calls the FashionCLIP-ResNet50 attribute recognition model to perform consistency checks on attributes that support automatic recognition, determining whether the category is still shirt, whether the color has changed to black, whether the collar type is still stand collar, whether the cuff type and pattern are consistent, and whether the fabric visual characteristics still conform to the woven fabric description. For complex quality rules such as length, placket type, size deviation, sewing quality, stripe / check matching, color difference grade, loose threads, and stains, the system does not declare them as fully automatic detection results. Instead, it generates manufacturer quality rule prompts and confirmation items based on the clothing knowledge graph, prompting users or quality inspectors to confirm in the interface; if necessary, these items are marked for manual review.

[0044] If the attribute consistency check fails, for example, the color fails to change to black or the neckline type drifts, the system will increase the redraw intensity by 0.05 and re-execute the mask constraint local redraw. The automatic retry limit is no more than 0.85, and the maximum number of automatic retryes is 3. When the attribute consistency check passes, or the check fails but the preset maximum number of automatic retryes is reached, the system will push the current best redraw result, attribute check report, Macondo "Twenty-Two No's" related prompts, and Q-SM woven appearance quality prompts to the user confirmation interface.

[0045] In the confirmation interface, users view a four-column attribute verification report, along with manufacturer quality rule prompts and confirmation items such as "no loose threads, no dirt or holes, no obvious color difference, symmetrical left and right, straight placket, no obvious abnormalities in stitching, and no defects in accessories." If the user chooses to accept the current result, the system outputs the final garment image and writes the image, enhanced prompts, negative prompts, masked areas, attribute verification results, and user acceptance records into the user preference sample library and the generated effect sample library. If the user points out problems such as uneven placket, obvious loose threads, or abnormal color differences, the system writes this feedback as a quality problem tag in the session log and generates a manufacturer quality rule pending review item. After manual review and confirmation, the manufacturer's personalized quality constraint sub-image is updated.

Claims

1. A method for knowledge-enhanced generation and local modification of mask constraints in a garment customization text-based image agent system, characterized in that: Includes the following steps: (1) Retrieval Enhancement Generation Based on Clothing Knowledge Graph: Construct a clothing knowledge graph containing clothing categories, clothing attributes, terminology mapping and manufacturer quality rule prompts, and establish a semantic retrieval index for the clothing knowledge graph; after receiving the user's natural language clothing generation request, execute the retrieval enhancement generation RAG process based on the semantic retrieval index and the clothing knowledge graph. The RAG process includes candidate clothing knowledge recall, constraint reasoning, candidate node reordering and prompt word enhancement; input the obtained standard attribute descriptions, manufacturer quality rule prompts and reference examples into the basic text-based image generation model to generate initial clothing images; (2) Mask-constrained local modification: Receive the user's natural language modification instruction for the initial clothing image, parse it to obtain the modification object, target attribute and retained attribute; generate a clothing instance-level or part-level spatial mask according to the modification object, and perform mask-constrained local redrawing of the initial clothing image based on the spatial mask; perform attribute consistency verification on the redrawing result, and generate a confirmation item based on the manufacturer's quality rule prompt, and display the attribute verification report, manufacturer's quality rule prompt and confirmation item to the user; output the final clothing image after user confirmation.

2. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 1, characterized in that, The construction of the clothing knowledge graph in step (1) includes the following sub-steps: (1.1) Hierarchical structure design: A three-level category hierarchy is established. The first level is the clothing category, including tops, bottoms, coats, dresses, and accessories; the second level is the clothing sub-categories, including shirts, T-shirts, suits, and jeans; the third level is the sub-categories, including formal shirts, casual shirts, and flannel shirts. General attributes and exclusive attributes are associated with each sub-category. General attributes include color, fabric, length, and pattern, while exclusive attributes include collar type, cuff type, and placket type. At the same time, a manufacturer-specific quality constraint sub-graph is established, which represents the manufacturer, manufacturer quality standard documents, appearance quality rules, sewing quality rules, size deviation rules, defect types, inspection items, and acceptance thresholds as graph nodes, and connects them to clothing category, attribute, and part nodes through the relationships of "define", "include", "constrain", and "applicable". (1.2) Terminology mapping and feature storage: For each node in the clothing knowledge graph, store the mapping relationship of standard terms, industry aliases, and user colloquialisms; for clothing category, attribute, part and defect type nodes with visual appearance features, call the FashionCLIP model to encode the standard term text description and several standard example images respectively, and take the average value of the resulting embedding vectors as the visual-semantic fusion feature vector of the node; for manufacturer, manufacturer quality standard document, inspection item, allowable deviation and acceptance threshold rule type nodes, store them in the vector database using text encoding or rule vector encoding method; (1.3) Bidirectional Index Construction: A bidirectional mapping relationship is established between the nodes of the clothing knowledge graph and the original text documents. The original text documents include general clothing terminology standards, enterprise garment quality standards documents, and appearance quality standards documents. The quality requirements, defect levels, appearance acceptance items, sewing requirements, allowable size deviations, and color difference requirements in the documents are structurally extracted and mapped to the corresponding clothing categories, parts, and attribute nodes. At the same time, the attribute legality constraint table and the manufacturer quality rule constraint table are maintained to record the legal value combinations of each attribute under each category and the quality acceptance conditions corresponding to specific manufacturers. These are used for constraint reasoning, candidate node clipping, prompt word construction, and manufacturer quality rule prompts in the RAG process. For complex quality items such as sewing quality, size deviation, thread ends, stains, and color difference levels that are difficult to automatically determine based solely on the stability of the generated image, the system converts them into generated prompt words, negative prompt words, and manufacturer quality rule prompts and confirmation items. If necessary, they enter the manual review or pending review process.

3. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 1, characterized in that, The RAG process in step (1) includes the following sub-steps: (1.4) User Intent Parsing and Semantic Retrieval: Receive user natural language input and extract clothing category keywords, attribute keywords, and manufacturer or manufacturer quality standard keywords; when the input contains a clear manufacturer, category, or manufacturer quality standard name, first limit the candidate category, attribute, and manufacturer quality rule range based on the clothing knowledge graph, then call the FashionCLIP text encoder to embed the user input and perform approximate nearest neighbor retrieval within the limited range; when the input is relatively ambiguous, first recall the Top-K semantically similar nodes in the vector database, then perform constraint reasoning and reordering based on the hierarchical structure of the clothing knowledge graph on the candidate nodes, and prioritize returning sub-category nodes that are consistent with the user intent in terms of category level, attribute set, manufacturer quality rules, and legal attribute combination; (1.5) Construction of enhanced search prompts: The retrieved standard terms, attribute constraint list, legal attribute combinations, manufacturer quality rule prompts, defect avoidance requirements and example image paths are assembled into a structured context; the enhanced prompts are constructed according to the template "[standard category name] + [attribute constraint list] + [manufacturer quality rule prompts] + [defect avoidance requirements] + [style reference description], solid color background, front view"; and complex quality items are output as negative prompts or manufacturer quality rule prompts and confirmation items; the example images are injected into the diffusion image generation model that supports image reference input in the form of image reference or IP-Adapter, guiding the generation results to align with the standard visual features and manufacturer quality rules in the clothing knowledge graph; (1.6) Incremental knowledge adaptation: When adding a new category, a new manufacturer, or a new manufacturer's quality standard document, the corresponding node information, attribute constraints, manufacturer quality rules, and standard example diagrams are entered into the clothing knowledge graph; FashionCLIP or a text encoder is called to encode the new node and its rule description and update the vector index. Without fine-tuning or retraining the basic text graph generation model, it can support incremental expansion of the knowledge configuration of new categories, new manufacturers, and manufacturer quality rules with few or low samples.

4. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 1, characterized in that, The local modification of the mask constraint in step (2) includes the following sub-steps: (2.1) Structured instruction parsing: Based on the clothing knowledge graph, the user's natural language modification instructions are parsed into a standardized JSON format, including the object to be modified, the list of modified attributes, and the list of retained attributes; the existence of the attributes in the parsing result is verified in the clothing knowledge graph; (2.2) Precise region mask generation: Call the clothing instance segmentation model to segment the region corresponding to the modified object from the initial clothing image and generate a pixel-level binary mask; when the modified object corresponds to the collar, cuff, placket, pocket, or hem, the part category output based on the instance segmentation model or combined with the SAM general segmentation model is used to refine the mask into a secondary mask, and then morphological dilation processing is performed. (2.3) Mask-constrained local redrawing: The style embedding vector and lighting parameters of the initial clothing image are extracted as constraints. Only the area within the mask is redrawn with diffusion. The number of redrawing steps is set to 20-30 steps, and the automatic initial redrawing intensity is set to 0.5-0.

7. A progressive redrawing strategy is adopted. The first 50% of the steps are used to modify the area within the mask, and the last 50% of the steps are used for edge blending and detail refinement. If automatic retry is triggered, the redrawing intensity increases by 0.05 each time, and the automatic retry limit does not exceed 0.

85. (2.4) Attribute-level consistency verification: The trained or fine-tuned FashionCLIP-ResNet50 attribute recognition model is invoked to predict the attributes that support automatic recognition in the initial clothing image and the redrawn clothing image respectively. The attributes that support automatic recognition include category, color, collar type, cuff type, pattern, fabric; length, placket type and other attributes not included in the prediction range of the attribute recognition model are processed as constraints of the clothing knowledge graph, local redraw prompts or user confirmation items. Among them, the content involving the manufacturer's quality rules is processed as manufacturer's quality rule prompts and confirmation items; verify whether the retained attributes are consistent and whether the modified attributes are successfully changed to the target value. If the verification fails, adjust the redraw intensity (increase by 0.05 each time, but not exceeding 0.85) and redraw. The maximum number of automatic retries is 3. (2.5) User Interaction Confirmation: When the attribute consistency check passes, or fails but reaches the preset maximum number of automatic retries, the system pushes the current optimal redraw result along with the attribute check report, manufacturer quality rule prompts, and confirmation items to the user confirmation interface. The attribute check report visually marks the check status (pass / fail) of the attributes that support automatic identification and compares the target value with the measured value. The manufacturer quality rule prompts and confirmation items list prompts for appearance defects, sewing quality, size deviation, color difference, stains, loose threads, or other manufacturer quality rule constraints, and marks items that cannot be reliably and automatically determined as user confirmation items. The system allows users to either accept the current result or undergo manual review. Users can choose one of the following actions: accept the current result and output the final garment image; manually adjust the redrawing intensity, inference steps, local redrawing prompts, or negative prompts to trigger a new round of redrawing; or abandon the current modification and retain the initial garment image. The system will write the user's acceptance, rejection, prompt correction, parameter adjustment, and quality issue feedback into the user preference sample library, the generated effect sample library, and the manufacturer's quality rule pending review library, respectively. Feedback involving changes to the manufacturer's quality standard document or the addition of defect rules will only enter the pending review process and will be written into the manufacturer's personalized quality constraint sub-graph after manual review and confirmation.

5. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 3, characterized in that, In step (1.4), the FashionCLIP text encoder is based on the CLIP-ViT-B / 32 backbone network and is pre-trained by comparative learning on the Fashion-200K and DeepFashion datasets. It can map clothing professional terms and general natural language descriptions to a unified semantic embedding space. The vector database is constructed using the FAISS library, and cosine similarity is used as the metric during retrieval. The Top-K value ranges from 3 to 5.

6. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 4, characterized in that, The clothing instance segmentation model in step (2.2) includes a trained or fine-tuned YOLOv8-Seg model with a morphological dilation processing pixel range of 3-5 pixels. It can also be combined with the SAM general segmentation model to perform secondary masking refinement on the neckline, cuffs, placket, pockets, and hem. The attribute recognition model in step (2.4) can use FashionCLIP-ResNet50. This model adds a multilayer perceptron classification head to the FashionCLIP visual encoder, supports the prediction of automatically identifiable attributes such as category, color, neckline type, cuff type, pattern, and fabric, and outputs the corresponding attribute prediction results and confidence scores. Length, placket type, and other attributes not included in the model prediction range are processed through clothing knowledge graph constraints, prompt word constraints, or user confirmation methods.

7. The knowledge enhancement generation and mask constraint local modification method for the clothing customization text image agent system according to claim 4, characterized in that, In step (2.5), the user confirmation interface is presented in a dialog-based interactive manner. The attribute verification report is displayed in a four-column table format with attribute name, original value, target value, and measured value, and the manufacturer's quality rule prompts and confirmation items are displayed simultaneously. When the user chooses to manually adjust the parameters and retry, the system provides four adjustable items: redraw intensity (range 0.5~0.9, step size 0.05), inference steps (range 20~50 steps), local redraw prompt words, and negative prompt words. The user's adjustment record is written to the session log for reference in subsequent parameter optimization.

Images