A shoe local design modification method and system based on text instructions

Abstract

The application provides a shoe local design modification method and system based on a text instruction, which comprises the following steps: obtaining an original shoe image and a natural language modification description; analyzing the description to construct a structured instruction containing a target component label and attributes; generating a component area mask based on the label, and calling corresponding domain adaptive feature parameters from a preset feature library; mapping the image to a latent space, and converting the instruction to a semantic vector; performing an iterative denoising process based on a diffusion architecture, during which the adaptive parameters are loaded and the mask is used for regional frozen calculation to realize accurate modification of the target area and keep the non-editing area unchanged; and finally decoding and reconstructing the output to obtain a modified target image. The method can modify and adjust the shoe local design by using a large language model, and can effectively improve the design efficiency.

Description

Technical Field

[0001] This invention relates to the field of footwear design technology, specifically to a method and system for modifying partial designs of footwear based on text instructions. Background Technology

[0002] In the field of modern footwear industrial design, designers often need to modify and adjust the details of existing shoe designs, such as changing the pattern on the midfoot area of ​​a shoe to a different pattern or shape. Currently, modifications to shoe designs still mainly rely on manual operation by designers, resulting in low efficiency. Summary of the Invention

[0003] The purpose of this invention is to overcome the aforementioned defects or problems in the prior art and to provide a method and system for modifying the partial design of footwear based on text instructions. This method can utilize a large language model to modify and adjust the partial design of footwear, which can effectively improve design efficiency.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: Technical Solution 1: A method for partial design modification of footwear based on text instructions, comprising the following steps: acquiring design input data, the design input data including an original footwear image to be modified and a natural language modification description for the original footwear image; performing semantic parsing and entity extraction operations on the natural language modification description to construct a structured modification instruction, the structured modification instruction including target component label data and target visual attribute parameters; performing visual region retrieval and pixel-level segmentation operations on the original footwear image according to the target component label data in the structured modification instruction to generate a component region mask indicating the target modification area; and, according to the structured modification instruction, retrieving the target component label data from a pre-built structured modification instruction. The system retrieves the domain-adaptive feature parameters associated with the target component label data mapping from the footwear component feature library; maps the original footwear image to the latent space to obtain initial latent feature data, and converts the structured modification instructions into semantic condition vectors; executes an iterative denoising calculation process based on a diffusion architecture, in which the domain-adaptive feature parameters are loaded into the calculation weights, the semantic condition vector is used as a guide, and the component region mask is used to perform region freezing calculation on the noise prediction results to obtain denoised target latent feature data; performs decoding and reconstruction operations on the target latent feature data, and outputs the locally modified target footwear image.

[0005] Technical Solution Two, based on Technical Solution One: The step of performing semantic parsing and entity extraction operations on the natural language modification description to construct a structured modification instruction includes: performing intent recognition calculation on the natural language modification description using a natural language processing algorithm; extracting entity words from the natural language modification description as target component label data, and extracting adjectives or noun phrases from the natural language modification description as target visual attribute parameters; combining the target component label data and the target visual attribute parameters into a key-value pair format data packet to obtain the structured modification instruction.

[0006] Technical Solution 3 based on Technical Solution 1: The step of performing visual region retrieval and pixel-level segmentation operations on the original shoe image according to the target component label data in the structured modification instruction to generate a component region mask indicating the target modification area includes: using the target component label data as search prompts, performing open-vocabulary target detection operations on the original shoe image to determine the bounding box coordinates of the target component; using the bounding box coordinates as prompt information, performing pixel-level segmentation operations on the image data within the bounding box coordinate range to generate a binary component region mask, wherein the numerical identifiers within the component region mask are the edited regions, and the numerical identifiers outside the component region mask are the non-edited regions.

[0007] Technical Solution 4 based on Technical Solution 1: The domain adaptation feature parameter is a low-rank adaptive weight matrix; before the step of executing the iterative denoising calculation process based on the diffusion architecture, it further includes: the diffusion architecture adopts a multimodal diffusion backbone network; the footwear component feature library stores multiple low-rank adaptive weight matrices pre-trained for different footwear components; according to the target component label data, the corresponding low-rank adaptive weight matrix is ​​read, and the low-rank adaptive weight matrix is ​​merged into the original weights of the attention projection layer of the multimodal diffusion backbone network.

[0008] Technical Solution 5 based on Technical Solution 4: The method further includes the step of performing spatial structure feature guidance: performing multi-scale convolution operation on the component region mask to extract multi-scale spatial feature data; and superimposing the multi-scale spatial feature data onto the corresponding layer features of the multimodal diffusion backbone network through residual connection to constrain the spatial structure in the iterative denoising calculation process.

[0009] Technical Solution Six based on Technical Solution One: The step of performing region freezing calculation on the noise prediction result using the component region mask includes: predicting first noisy latent feature data based on the latent features of the current step number during each step of the denoising iteration calculation; performing forward noise addition calculation on the initial latent feature data to obtain second noisy latent feature data matching the current step number; performing weighted fusion calculation on the first noisy latent feature data and the second noisy latent feature data using the component region mask; retaining the first noisy latent feature data in the edit area indicated by the component region mask; replacing it with the second noisy latent feature data in the non-edit area indicated by the component region mask; obtaining fused latent feature data for the next iteration calculation.

[0010] Technical Solution Seven based on Technical Solution One: The method for constructing the footwear component feature library includes: performing batch image generation operations to obtain synthetic image data containing footwear products and corresponding synthetic line drawing data; determining the mask annotation data of each footwear component in the synthetic image data based on attention map extraction algorithms or manual annotation operations, and constructing a triplet training dataset containing text descriptions, synthetic line drawings, and component masks; using the triplet training dataset to perform fine-tuning training calculations on the basic diffusion model to obtain feature parameters that can reconstruct specific footwear components, and storing the feature parameters in the footwear component feature library.

[0011] Technical solution eight based on technical solution four: The multimodal diffusion backbone network contains multiple computational layers; the low-rank adaptive weight matrix is ​​only merged into a specific intermediate layer interval in the multimodal diffusion backbone network that is responsible for semantic and structural combination, and the intermediate layer interval is the fifteenth to the forty-fourth layer.

[0012] Technical Solution Nine based on Technical Solution One: The method further includes: displaying the target shoe image through a display interface; when a further modification instruction for the target shoe image is received, defining the target shoe image as a new original shoe image, and repeatedly executing the parsing, retrieval, segmentation, mapping, and iterative denoising calculation process.

[0013] Furthermore, this invention also provides technical solution ten: A system for partial design modification of footwear based on text instructions, characterized in that it includes: an input unit for acquiring design input data, the design input data including an original footwear image to be modified and a natural language modification description for the original footwear image; an instruction parsing unit for performing semantic parsing and entity extraction operations on the natural language modification description to construct a structured modification instruction, the structured modification instruction including target component label data and target visual attribute parameters; a visual computing unit for performing visual region retrieval and pixel-level segmentation operations on the original footwear image according to the target component label data in the structured modification instruction, generating a component region mask indicating the target modification region; and a resource scheduling unit. The system is used to retrieve domain-adaptive feature parameters associated with the target component label data mapping from a pre-built footwear component feature library according to the structured modification instructions; the denoising generation unit is used to map the original footwear image to the latent space to obtain initial latent feature data, and convert the structured modification instructions into semantic condition vectors; it executes an iterative denoising calculation process based on a diffusion architecture, in which the domain-adaptive feature parameters are loaded into the calculation weights, the semantic condition vector is used as a guide, and the component region mask is used to perform region freezing calculation on the noise prediction results; finally, the calculated target latent feature data is decoded and reconstructed to output the locally modified target footwear image.

[0014] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects: Technical Solution 1 simplifies the process of modifying partial designs of footwear by introducing a large language model. Designers only need to provide text instructions with clear modification information, and the large language model can automatically generate and output images of the modified partial designs, thereby improving the efficiency of footwear design modification. Furthermore, this solution overcomes the problem of insufficient semantic understanding of footwear components in existing text-guided image editing technologies. Existing conventional text-guided image editing technologies typically input natural language instructions as a whole semantic condition into a diffusion architecture for probability calculation. Because general models lack precise feature correspondences for specific fine-grained components in industrial design (such as "ear loops" and "midsoles" in footwear), this holistic semantic processing method often fails to accurately converge the semantic focus to the target object when facing local modification needs for specific components. This leads to the calculation process altering the overall style of the image or incorrectly modifying irrelevant areas. Conventional solutions often rely on manually drawn masks to limit the generation range. This is not only inefficient in data processing but also only provides physical isolation in space, failing to fundamentally solve the problem of insufficient semantic understanding of specific industrial components by the computational model. Consequently, the texture or structure within the generated area may still deviate from industrial design standards.

[0015] This scheme first parses the unstructured natural language description into structured key-value pairs containing target component labels and visual attribute parameters. Using these as indexes, it automatically retrieves the corresponding domain-adaptive feature parameters from a pre-built library before the computation process begins. In subsequent iterative denoising computations, these retrieved parameter data are dynamically loaded and merged into the weight matrix of the attention layer. This data processing method allows a general diffusion computation architecture to temporarily acquire the ability to process specific industrial components within a given computational task without altering its original infrastructure. Compared to conventional methods that rely solely on text-based conditional vectors for generation, this parameter injection mechanism based on structured data enhances the semantic response accuracy to specific design objects from the computational level, ensuring that the generated components conform to specific design intents in terms of geometric structure and texture details. Furthermore, in each step of the iterative denoising computation, a region freezing computation based on component region masks is performed. Specifically, this step introduces a data stream that performs forward noise addition computation on the original image, weightedly fusing the original image noise data calculated at the current time step with the denoised data predicted by the model based on mask tensors. This mathematical constraint in the latent feature space forces the feature evolution trajectory of the non-edited region to be strictly locked within the signal distribution of the original image, while only the edited region is allowed to evolve in a directional manner according to the injected semantic parameters. Since both undergo a complete denoising iteration calculation in the same latent space coordinate system, the final decoded image exhibits mathematical consistency in light and shadow projection, texture transition, and perspective relationships, avoiding the splicing marks or background distortions commonly found in conventional local repairs.

[0016] In technical solution two, unstructured descriptive text is converted into key-value pair format data packets by using natural language processing algorithms. Ambiguous modifiers are precisely anchored to the attribute values ​​of specific fields, so that subsequent calculation processes can directly read the definite parameters instead of making probabilistic inferences about the whole sentence. This avoids the instruction parsing deviation caused by differences in language habits and ensures that the input data for subsequent retrieval and location calculations has a high degree of standardization.

[0017] In technical solution three, a cascaded computation strategy combining open vocabulary detection and pixel-level segmentation is adopted. The bounding boxes determined by object detection are used as spatial prior information, significantly reducing the search space for segmentation operations. This allows the segmentation algorithm to perform fine-grained pixel classification only within a local area. This coarse-to-fine computational logic automatically obtains accurate topological masks for the irregular edges of footwear components without manual intervention, providing high-precision boundary constraints for subsequent region freezing calculations.

[0018] In technical solution four, a pre-trained low-rank matrix is ​​loaded into the computational weights of the attention projection layer. Compared with full parameter fine-tuning or simple prompt word guidance, this processing method, which only changes the value of a specific projection matrix, changes the vector mapping relationship in the feature space with a very small parameter increment. This enables the computational architecture to accurately respond to the semantic features of specific footwear components, while maintaining the backbone network's ability to process general image features. This achieves flexible adaptation to a variety of different industrial components under a single computational architecture.

[0019] In technical solution five, convolution operations and residual superposition of multi-scale spatial features are introduced. The stability of the edge contour and internal lines of the component is further enhanced by the guidance of spatial features. This ensures that when the material or color attributes of the component are changed significantly, the original design lines and structural proportions of the component will not be deformed or collapsed due to the randomness of the generation process.

[0020] Technical solution six also includes a forward noise addition calculation process for the original image. Compared with directly cutting in the pixel space or using mismatched noise data for filling, this weighted fusion calculation based on isomorphic noise distribution enables the edited area and the non-edited area to maintain the consistency of illumination and the smoothness of pixel transition during final decoding, eliminating boundary stitching artifacts caused by noise level mismatch.

[0021] In technical solution seven, a strongly aligned dataset containing text, line art, and masks is constructed through synthetic data generation and attention map extraction. This data construction method solves the problem of scarce and difficult annotation of real industrial design data, enabling the trained parameters to learn the strong correlation between "semantic description-geometric contour-spatial location" rather than just learning pixel reconstruction of images, thereby improving the decoupling ability and response accuracy of feature parameters when faced with complex text instructions.

[0022] In technical solution eight, the weight modification is restricted to the intermediate level range between the fifteenth and forty-fourth layers. By locking the intermediate level for parameter intervention, the semantic and structural combination process of the image object components is accurately captured. This avoids disrupting the general image composition rules at the lower level and also prevents excessive interference with the global style consistency of the higher level, thus enabling independent modification of the attributes of specific components.

[0023] Technical Solution Nine establishes an iterative calculation mechanism based on image state updates. This mechanism allows the output of the previous calculation to be used as the initial state input for the next calculation, supporting the need for continuous modifications during the design process. By continuously updating the original image baseline, users can overlay new design changes while retaining the results of previous modifications, enabling this data processing flow to adapt to complex multi-round iterative optimization scenarios in industrial design.

[0024] Technical solution ten provides a text-based shoe design modification system. This system integrates an input unit, an instruction parsing unit, a visual computing unit, a resource scheduling unit, and a noise reduction and generation unit. It replaces the manual operation workflow in traditional design methods, realizes quick operation of shoe design modification, and effectively improves the efficiency of shoe design modification. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments are briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the method for modifying a part of a shoe design based on text instructions, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of a text-based shoe design modification system according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are preferred embodiments of the present invention and should not be considered as excluding other embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] The illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0029] Unless otherwise expressly defined, the use of terms such as "first," "second," or "third" in the claims, description, and accompanying drawings of this invention is for distinguishing different objects and not for describing a specific order.

[0030] In the claims, description and accompanying drawings of this invention, the terms "comprising," "having," and variations thereof are used to mean "including but not limited to."

[0031] Reference Figure 1The diagram illustrates a flowchart of a method for modifying a part of a shoe design based on text instructions, according to an embodiment of the present invention. Also, referring to... Figure 2 The diagram illustrates a module schematic of a text-instruction-based shoe partial design modification system according to an embodiment of the present invention.

[0032] The system mainly includes the following modules: The input unit is used to acquire design input data, which includes the original shoe image to be modified and a natural language description of the modification to the original shoe image. The instruction parsing unit is used to perform semantic parsing and entity extraction operations on the natural language modification description, and to construct a structured modification instruction. The structured modification instruction includes target component label data and target visual attribute parameters. The visual computing unit is used to perform visual region retrieval and pixel-level segmentation operations on the original shoe image based on the target component label data in the structured modification instruction, and generate a component region mask that indicates the target modification area. The resource scheduling unit is used to call the domain-adaptive feature parameters associated with the target component label data mapping from the pre-built footwear component feature library according to the structured modification instructions. The denoising generation unit maps the original shoe image to the latent space to obtain initial latent feature data and transforms the structured modification instructions into semantic condition vectors. It executes an iterative denoising calculation process based on a diffusion architecture. In the iterative denoising calculation process, the domain-adaptive feature parameters are loaded into the calculation weights. The semantic condition vector is used as a guide, and the region freezing calculation is performed on the noise prediction results using the component region mask. Finally, the target latent feature data obtained by calculation is decoded and reconstructed to output the locally modified target shoe image.

[0033] This system is used to implement the aforementioned method. The specific steps of the method will be described in detail below, focusing on the specific modules of the system. This detailed explanation will enable those skilled in the art to clearly understand the specific structure of the system. Furthermore, after understanding the method, those skilled in the art will be able to implement the system based on common knowledge in the field.

[0034] like Figure 1 As shown, the method mainly includes the following steps: S100, Construct a feature library for footwear components; In this embodiment, in order to overcome the problem that the general large model has insufficient semantic understanding of footwear special components (such as lugs, midsoles, eyelets, etc.), the system pre-constructs a feature parameter library containing domain-specific knowledge.

[0035] S100 further includes the following sub-steps: S101, Perform batch image generation operation to obtain composite image data containing footwear products and corresponding composite line drawing data; In this step, a pre-trained base model with high-fidelity generation capabilities (such as Qwen-Image or Stable Diffusion XL) is used as the data generator. Specifically, the system has a pre-built library of footwear design cue templates, which uses a combination of "subject + component + attribute + style". The system automatically fills in variables to generate specific instructions, such as cue words like "white chunky sneakers, red pull tabs, clear outline, industrial design style" or "black canvas shoes, yellow rubber midsole, minimalist lines". The model generates corresponding high-resolution synthetic images based on these cue words, with a resolution that can be set to 1024 x 1024 pixels. Simultaneously, to increase data diversity and enhance the model's understanding of geometric structures, the system also performs edge detection processing on the synthetic images, such as using the Canny algorithm or HED edge detection algorithm, to obtain black and white synthetic line drawings that are strictly aligned with the synthetic images. In this way, the system can generate a large amount of high-quality synthetic sample data in a short time.

[0036] S102, based on attention map extraction algorithm or manual annotation operation, determine the mask annotation data of each footwear component in the synthetic image data, and construct a triplet training dataset containing text description, synthetic line drawing and component mask; To enable the model to accurately perceive the spatial location of specific components in an image, this embodiment employs an automated spatial attribution method. During the generation of the diffusion model, the system records the activation map of the cross-attention layer. The principle is that when the model generates based on the prompt "red ear," the "ear" token in the text encoder interacts with the spatial features of the image. The system extracts the attention weight matrix corresponding to this token, which has a high response value in the pixel region where the "ear" is located. By setting a numerical threshold, such as 0.4, the system transforms the attention weight matrix into a binary component region mask, where regions with a value of 1 represent components and regions with a value of 0 represent the background. For some structurally complex components, such as overlapping shoelaces, the system supports manual verification to correct the automatically generated mask, ensuring accuracy. The final training dataset is stored in a structured format. Each training sample contains a text description, an image or line drawing path, and the corresponding component mask path, forming a triplet dataset. An example of its data structure is shown below: { "sample_id": "SHOE_SYN_00124", "text_description": "A white running shoe viewed from the side, with a fluorescent green heel counter", "image_data_path": ". / data / images / 00124.png","sketch_data_path": ". / data / sketches / 00124.png", "component_masks": [ { "label": "heel_counter", "mask_path": ". / data / masks / 00124_heel.png"} ]} S103 uses the triplet training dataset to perform fine-tuning training calculations on the basic diffusion model, obtains feature parameters that can reconstruct specific footwear components, and stores the feature parameters in the footwear component feature library.

[0037] In this step, the system does not update all parameters of the base model, but instead uses low-rank adaptation technology for efficient fine-tuning. The specific training configuration is as follows: weight updates are performed only on the attention projection layer in the multimodal diffusion backbone network, focusing on adjusting the mapping relationship between the query matrix and the key-value matrix. In terms of parameter settings, the rank of the low-rank matrix is ​​set to 64, and the scaling factor is set to 128. This parameter choice balances feature learning capability with storage file size. After training, the system extracts the updated low-rank adaptation weight matrix and saves it as a lightweight feature file. The footwear component feature library is organized and stored in key-value pairs, supporting quick retrieval via component tags. For example, when the key is "ear flap," the corresponding value is the storage path of the ear flap feature file; when the key is "midsole," the corresponding value is the storage path of the midsole feature file. In this way, the feature library becomes a dynamically loadable external knowledge base, allowing the system to read and load different feature files in real time according to subsequent instruction requirements.

[0038] S110, Obtain design input data, which includes the original shoe image to be modified and a natural language description of the modifications to the original shoe image; This step is performed by the input unit in the system, aiming to establish a data entry point for human-computer interaction and to perform standardized preprocessing of the raw data. Specifically, the input unit receives the original shoe image to be modified uploaded by the user through a graphical user interface or application programming interface. This image can be a real photograph of the shoe product or a design sketch rendered by 3D modeling software. To adapt to the input specifications of the subsequent generation engine, the input unit performs a series of preprocessing operations on the received image data. First, the input unit converts image files of different formats, such as the Joint Image Experts Group format or portable network graphics format, into a unified red, green, and blue three-channel color image array. Subsequently, the input unit performs size normalization processing, using a bicubic interpolation algorithm to adjust the image resolution to the system's preset standard size, such as 1024×1024 pixels. On this basis, the input unit further normalizes the image pixel values, mapping the integer values ​​originally distributed in the range [0, 255] to the floating-point range [-1, 1], thereby obtaining tensor data that meets the input distribution requirements of the diffusion model.

[0039] While acquiring image data, the input unit also simultaneously receives a natural language description of the modifications the user inputs for that image. This description directly reflects the user's design intent and typically includes the specific object to be modified and the desired visual effect. For example, a user might input "change the pull tabs of this shoe to red leather" or "change the sole to a semi-transparent rubber outsole." The input unit performs basic cleaning on the received text string, removing leading and trailing whitespace and invisible control characters, and validating its length. Finally, the input unit packages the preprocessed image tensor data and the cleaned text string data into a set of associated design input data pairs, which are then transmitted to the system's instruction parsing unit for use in subsequent steps.

[0040] S120 performs semantic parsing and entity extraction operations on the natural language modification description to construct a structured modification instruction, which includes target component label data and target visual attribute parameters; This step is performed by the instruction parsing unit in the system. Its core task is to transform the user's colloquial and ambiguous instructions into standardized parameters that the computer can strictly execute. The instruction parsing unit integrates a natural language processing algorithm module, which can understand the professional terminology and syntactic structures in the field of industrial design.

[0041] S120 further includes the following sub-steps: S121, Use natural language processing algorithms to perform intent recognition calculation on the natural language modified description; In this sub-step, the instruction parsing unit first loads a pre-built semantic analysis model. This model can be a lightweight large language model based on a transformer architecture, such as the Qwen model with 7 billion parameters, or a bidirectional encoder representation model fine-tuned by instructions. The instruction parsing unit inputs the received text string into the model, performing word segmentation and word vector encoding operations. The model identifies the user's core operational intent by analyzing verb and preposition structures in the text. For example, when the input text is "change the ear to red", the model identifies the operation type as "modify"; when the input text is "remove the sign at the heel", the model identifies the operation type as "delete". The system determines that the current task belongs to a local attribute editing task through the probability output of the classification layer, thereby activating the subsequent entity extraction process.

[0042] S122, extract entity words from the natural language modification description as target component label data, and extract adjectives or noun phrases from the natural language modification description as target visual attribute parameters; In this sub-step, the instruction parsing unit uses named entity recognition technology to fill in the text slots. The system predefines two key slots: the target object slot and the target attribute slot. The instruction parsing unit traverses the input word vector sequence, calculating the probability that each word belongs to a specific slot. For example, in the instruction "modify the pull tab of this shoe to fluorescent green leather," the algorithm identifies "pulls tab" as an entity word and extracts it as the target component label data; simultaneously, it identifies "fluorescent green" and "leather material" as modifying adjectives or noun phrases and extracts them as target visual attribute parameters. In particular, the instruction parsing unit also has a terminology normalization function, which can use a pre-built thesaurus to map non-standard terms entered by the user to standard industrial terms. For example, if the user enters "shoehorn" or "heel pull tab," the system will uniformly convert them into the standard target component label data "pulls tab" according to the mapping relationship to ensure the accuracy of subsequent retrieval steps.

[0043] S123, combine the target component label data and target visual attribute parameters into a key-value pair format data packet to obtain a structured modification instruction.

[0044] In this sub-step, the instruction parsing unit encapsulates the extracted standardized data into a machine-readable structured format, such as JavaScript object notation. This data package explicitly defines the various parameter dimensions of the modification operation, eliminating the ambiguity of natural language. The final generated structured modification instruction is then passed to the subsequent visual computing unit and resource scheduling unit. An example of the data content of a typical structured modification instruction is as follows: {"task_id": "EDIT_REQ_20251013_001","operation_type": "modification","target_entity": {"raw_text": "pull tab","standard_label": "pull_tab" / / Normalized target component label data},"target_attributes": {"color": "fluorescent green", / / Target visual attribute parameter"material": "leather" / / Target visual attribute parameter},"context_constraints": []} S130, based on the target component label data in the structured modification instruction, perform visual region retrieval and pixel-level segmentation operations on the original shoe image to generate a component region mask indicating the target modification area; This step is performed by the visual computing unit in the system and aims to solve the spatial localization problem of "point-and-click modification". The visual computing unit receives the original shoe image from the input unit and standardized target component label data (e.g., "pull_tab") from the instruction parsing unit. Due to the complex background of industrial design images and the highly irregular shapes of components, the visual computing unit adopts a cascaded computing strategy of "coarse localization followed by fine segmentation" to convert semantic labels into binary spatial tensors with the same resolution as the image.

[0045] S130 further includes the following sub-steps: S131, using the target component label data as search prompts, perform open vocabulary target detection on the original shoe image to determine the bounding box coordinates of the target component; In this sub-step, the visual computing unit invokes a pre-built open-vocabulary object detection network (e.g., the GroundingDINO model). Unlike traditional closed-set detection models that can only detect fixed categories, this network possesses cross-modal language understanding capabilities. The visual computing unit inputs the target part label data as text prompts into the network. The network calculates a cross-modal attention map of text features and image features to retrieve the region in the original shoe image that best matches the semantics of the label. The detection result is output in the form of bounding boxes, specifically represented by a set of coordinate parameters. This coordinate system defines a rectangular bounding box containing the target component. For example, for the label "pulling loop," the network would output the coordinates of a rectangle tightly enclosing the pull tab area on the heel. While this step cannot precisely depict the edge contour of the component, it provides a strong spatial prior for subsequent segmentation calculations, significantly narrowing the search range and eliminating background interference.

[0046] S132, using the bounding box coordinates as prompt information, performs pixel-level segmentation operation on the image data within the bounding box coordinate range to generate a binary component region mask, where the numerical identifiers within the component region mask are the edited regions, and the numerical identifiers outside the component region mask are the non-edited regions.

[0047] In this sub-step, the visual computing unit performs refinement using an image segmentation model (e.g., the Segment Anything Model, SAM). The system then processes the bounding box coordinates obtained in the previous step. Geometric cues are input to the cue encoder of the segmentation model, while the original image is input to the image encoder. The segmentation model uses feature information within the bounding box to infer the fine contours of the foreground object and automatically removes background pixels (such as shoe background or shadows) within the bounding box, thereby generating a high-precision segmentation result. The visual computing unit formats this segmentation result into a binary matrix. Its resolution is comparable to the original image's high resolution. Hekuan Maintain consistency. In this matrix, the pixel positions belonging to the target component (such as the lug) are assigned values. The area marked as the editing area is used; all other pixel positions are assigned a value. This is marked as a non-editable area. The final generated part region mask can accurately cover the irregular edges of the target part, providing a pixel-level spatial constraint benchmark for the region freezing calculation in the subsequent generation process.

[0048] S140, according to the structured modification instructions, call the domain adaptation feature parameters associated with the target component label data mapping from the pre-built footwear component feature library; This step is executed by the resource scheduling unit in the system, whose core function is to enable on-demand expansion of the generative model's capabilities. The resource scheduling unit receives target component label data from the instruction parsing unit and uses it as a retrieval key to search within the system's pre-built footwear component feature library. This feature library acts as an external knowledge storage for the model, allowing the general model to temporarily acquire the ability to recognize specific industrial components by attaching external parameters without retraining the entire network.

[0049] S140 further includes the following sub-steps: S141, Determine the multimodal diffusion backbone network used in the iterative denoising calculation process. The backbone network contains multiple computational levels. In this sub-step, the resource scheduling unit first determines the infrastructure on which the currently generated task is based. This embodiment uses a multimodal diffusion backbone network as the core computing engine. This network is built on a converter architecture and contains billions of parameters. Structurally, this backbone network consists of a series of stacked converter modules, which are arranged sequentially to form multiple computing layers, typically labeled as the [number]th [level]. Layer to the first Layers. Each computational layer contains a multi-head self-attention mechanism, a cross-attention mechanism, and a feedforward neural network, responsible for handling pixel correlations within the image, cross-modal interactions between text and image, and nonlinear transformations of features, respectively. Confirming the architecture and layer definition of the backbone network is the physical basis for subsequent parameter injection.

[0050] S142, Based on the target component label data, read the corresponding low-rank adaptive weight matrix from the footwear component feature library; In this sub-step, the resource scheduling unit performs index matching in the footwear component feature library based on the target component label data, such as "ear flap," locates the corresponding feature file path, and loads it into memory. This feature file stores a pre-trained low-rank adaptation weight matrix for that component. The low-rank adaptation technique increments the parameters... Decomposed into two low-rank matrices and The product of, i.e. This significantly reduces parameter storage requirements. The resource scheduling unit reads the data from these two low-rank matrices, preparing to apply them to specific weights in the backbone network. Compared to loading the entire model file, reading these lightweight low-rank matrices requires minimal system overhead, ensuring the system's real-time response speed.

[0051] S143, the low-rank adaptive weight matrix is ​​merged into the original weights of the attention projection layer of the multimodal diffusion backbone network; in particular, the low-rank adaptive weight matrix is ​​only merged into the specific intermediate layer intervals responsible for semantic and structural combination in the multimodal diffusion backbone network, namely layers 15 to 44.

[0052] In this sub-step, the resource scheduling unit performs the physical injection of parameters. The system does not modify all parameters of the backbone network, but rather updates precisely the query projection matrix and numerical projection matrix in the attention mechanism. The update formula is as follows: ,in For the original weights, The combined weights, This is the scaling factor. More importantly, this embodiment implements a strict hierarchical interval control strategy. The resource scheduling unit only merges the low-rank adaptive weight matrix into a specific interval from layer 15 to layer 44 of the backbone network. Experiments show that the lower layers of the network are mainly responsible for processing basic textures and geometric lines, the higher layers are mainly responsible for processing abstract styles and global semantics, and the middle interval from layer 15 to layer 44 is the key area for deep alignment of semantic concepts and spatial structures. By injecting footwear component knowledge only into this interval, the system ensures that the model can accurately reconstruct the professional structure of "ears" or "midsole" while avoiding damage to the original basic image quality or excessive interference with the overall style of the image, achieving the best balance between local modification and overall effect.

[0053] S150 maps the original shoe image to the latent space to obtain initial latent feature data and transforms the structured modification instructions into semantic condition vectors; This step is performed by the denoising calculation unit in the system, aiming to unify pixel-level and text-level input data into an internal feature representation that can be processed by the diffusion generation model. Since performing diffusion calculations directly on high-resolution images would consume enormous computational resources, this embodiment adopts a latent diffusion strategy to perform feature evolution in a low-dimensional latent space.

[0054] First, for the original shoe image, the denoising unit calls the encoder module in a pre-built variational autoencoder. This encoder is a pre-trained convolutional neural network capable of compressing and mapping the image from pixel space to a latent feature space. Specifically, the encoder receives images of a size of [size missing]. Furthermore, the original image tensor, containing red, green, and blue color channels, is used to extract the perceptual feature distribution of the image through layer-by-layer convolution and downsampling operations, ultimately outputting a dimensionless tensor. The multi-channel feature tensor, i.e., the initial latent feature data, significantly reduces the amount of data while fully preserving the original shoe's structural outline, light and shadow distribution, and texture details, providing a compact and efficient data foundation for subsequent denoising and reconstruction.

[0055] Concurrently, for structured modification instructions, the denoising computation unit performs semantic vectorization transformation using a text encoder. The denoising computation unit first reads the target component label data (e.g., "ear") and target visual attribute parameters (e.g., "red leather") from the structured modification instructions, reconstructing them into a descriptive cue word sequence conforming to natural language grammar. Subsequently, a text encoder based on a transformer architecture (e.g., a contrastive language image pre-trained model or the text encoding branch of a T5 model) performs word segmentation on this sequence, transforming the text into a tokenized sequence. The text encoder further maps these tokens into a high-dimensional embedding vector sequence, i.e., a semantic conditional vector. This vector is rich in semantic information about the modification target and will guide the model to generate visual content consistent with the description during subsequent diffusion through a cross-attention mechanism.

[0056] S160, execute the iterative denoising calculation process based on diffusion architecture. In the iterative denoising calculation process, the domain adaptation feature parameters are loaded into the calculation weights, the semantic condition vector is used as a guide, and the component region mask is used to perform region freezing calculation on the noise prediction results to obtain the denoised target potential feature data. This step is executed by the denoising computation unit in the system and is the core computational component of the entire technical solution. Based on a pre-trained diffusion model architecture, the denoising computation unit constructs a discrete time-step iterative process, typically set to 50 or 100 steps. In each step, the unit utilizes a backbone network loaded with domain-adaptive feature parameters, guided by semantic conditional vectors, to perform denoising prediction on the current latent features.

[0057] S160 further includes the following sub-steps: In this sub-step, the denoising computation unit first activates the spatial structure guidance network, which runs parallel to the backbone network. This network receives the binarized part region mask from the visual computation unit and performs a series of downsampling and convolution operations on it. Through this process, the network extracts the spatial geometric features of the mask at different resolution levels, i.e., multi-scale spatial feature data. This feature data not only contains the edge contour information of the part but also implicitly contains the topological location information of the part in the image. Subsequently, the denoising computation unit uses a zero-convolution strategy to stack these multi-scale spatial feature data layer by layer into the corresponding layers of the encoder module or intermediate module of the multimodal diffusion backbone network through residual connections.

[0058] S162, when performing each step of the denoising iteration calculation, predict the first noisy latent feature data based on the latent features of the current step number.

[0059] In this sub-step, the denoising computation unit drives the multimodal diffusion backbone network to perform one forward inference. Input data includes the current time step. The model outputs latent features, time-step embedding vectors, semantic conditional vectors generated in S150, and spatial features injected in S161. At this point, the attention projection layer of the backbone network has been loaded with a specific low-rank adaptive weight matrix as described in S140, enabling the model to accurately understand the descriptions of specific components such as "ears" in the semantic conditional vectors. The model outputs a tensor with the same dimension as the latent features, namely the first noisy latent feature data. This data represents the noise components that the model predicts should be removed in the current state based on the text instructions and domain knowledge, or, in some sampling algorithms, the predicted noise components. Estimated value.

[0060] S163, perform forward noise addition calculation on the initial latent feature data to obtain the second noisy latent feature data that matches the current step number.

[0061] In this sub-step, the denoising calculation unit utilizes the Markov chain property of the diffusion model to perform forward denoising calculation on the initial latent feature data representing the original image generated in S150. Specifically, the system calculates based on the current denoising step number. Based on a preset noise scheduling table, the system calculates the corresponding signal-to-noise ratio (SNR) parameter. Then, the system samples random noise from a standard normal distribution and superimposes it onto the initial latent feature data at a specific ratio, thus obtaining the second noisy latent feature data. This data is mathematically equivalent to: without any modification to the original image, its value at the current time step... The noise state that should be presented. This step provides a reference signal for subsequent background freezing.

[0062] S164, using the component region mask, perform a weighted fusion calculation on the first noisy latent feature data and the second noisy latent feature data. The first noisy latent feature data is retained in the editing area indicated by the component region mask, and replaced with the second noisy latent feature data in the non-editing area indicated by the component region mask, to obtain the fused latent feature data for the next iteration calculation.

[0063] In this sub-step, the denoising calculation unit performs the core region freezing operation. The system utilizes component region masks. As a mixing coefficient, the first noisy latent feature data predicted by the model. Second noisy latent feature data of the original image Perform the Hadamard product operation (element-level multiplication). The formula can be expressed as follows: Through this calculation, within the edit region where the mask value is 1, the system's denoising path is entirely dominated by model predictions containing the intention to modify; while in the non-edit region where the mask value is 0, the system's denoising path is forcibly reset to the noise trajectory of the original image. This fused data... The sampled data (such as a DPM-Solver or Euler sampler) is fed into a sampler to calculate the next time step. The system ensures that the final generated image maintains strict pixel-level consistency with the original image in terms of background, lighting, and non-target areas by repeating this operation in each iteration.

[0064] S170, performs decoding and reconstruction operations on the target latent feature data, and outputs a locally modified image of the target shoe; This step is primarily executed by the denoising calculation unit within the system, aiming to restore the abstract features generated by the diffusion model to a visually perceptible image. Once the iterative denoising calculation process in S160 is complete, for example, after completing a preset 50 sampling steps, the system obtains clean target latent feature data with all noise components removed. This data is currently still in a low-dimensional latent space. Although it contains all the information of the modified "red pull tab" and the retained original shoe background, its data format is a compressed multi-channel tensor, making it unviewable directly.

[0065] The denoising unit invokes the decoder module within the pre-defined variational autoencoder to perform decoding and reconstruction operations on the latent feature data of the target. The decoder is a neural network symmetrical to the encoder structure in the S150, primarily composed of a series of deconvolutional layers and upsampling layers. The decoder's receiving dimension is... The target latent feature data is used to gradually restore the data dimension to the original spatial resolution through layer-by-layer feature expansion and nonlinear transformation, for example... In this process, the decoder reconstructs the subtle textures of the shoe upper material, the reflective details of light shining on the leather, and the color gradations of the background fabric based on semantic cues in the latent features.

[0066] After completing the decoding operation, the denoising calculation unit obtains numerical values ​​distributed in The system performs further post-processing operations, denormalizing the tensor data and mapping the values ​​back to the range. The system calculates and truncates standard pixel values ​​to ensure validity. It then converts the data into a standard image file format, such as a three-channel bitmap. Finally, the system presents the modified target shoe image to the user through a graphical user interface for designer review. In this image, the target components (such as the lugs) have been modified according to the instructions, while the remaining areas (such as the sole, laces, and background) maintain strict pixel-level consistency with the original image.

[0067] S180: When a further modification instruction for the target shoe image is received, the target shoe image is defined as a new original shoe image, and steps S120 to S170 are repeated.

[0068] First, the system displays the partially modified target shoe image output from S170 in full screen through a graphical user interface, providing zoom and pan tools so users can examine the texture details and edge blending of the modified area in detail. At this point, the system is in an interactive waiting state. If the user is satisfied with the current design, they can choose to save or export the image; if the user feels further adjustments are needed, such as finding the color too dark or wanting to modify another component in the image, such as "changing the midsole to black," the user can continue to enter new natural language modification descriptions in the text input boxes on the interface.

[0069] Once the input unit receives this further modification instruction, the system immediately triggers the iterative logic. At the data level, the system no longer uses the initially imported original image as a baseline, but instead redefines the target shoe image that has just been generated and completed the previous round of modifications as the "original shoe image" for the next round of calculations. This state update operation ensures that subsequent modifications are based on the latest design state, thus supporting overlay design changes.

[0070] Subsequently, the system automatically restarts the calculation process, sequentially executing the semantic parsing in S120, visual region retrieval and segmentation in S130, resource scheduling in S140, data encoding in S150, and iterative denoising calculation in S160, based on the new original shoe image and new modification instructions. For example, in the second round of modifications, the system identifies new target components (such as the midsole), automatically generates new component region masks, and calls the domain adaptation feature parameters corresponding to the midsole. Through this iterative mechanism, users can gradually complete the fine-tuning of each component of the entire shoe through continuous natural language dialogue until a final satisfactory design is obtained.

[0071] This embodiment relates to a text-based instruction-based method and system for modifying partial shoe designs. By introducing a large language model, it simplifies the process of modifying partial shoe designs. Designers only need to provide text instructions with clear modification information, and the large language model can automatically generate and output images of the modified partial shoe designs, thereby improving the efficiency of shoe design modification. Furthermore, this solution overcomes the problem of insufficient semantic understanding of shoe components in existing text-guided image editing technologies. Existing conventional text-guided image editing technologies typically input natural language instructions as a whole semantic condition into a diffusion architecture for probability calculation. Because general models lack precise feature correspondences for specific fine-grained components in industrial design (such as "ear loops" and "midsoles" in footwear), this holistic semantic processing approach often fails to accurately converge the semantic focus to the target object when facing local modification needs for specific components. This results in the calculation process altering the overall style of the image or incorrectly modifying irrelevant areas. Conventional solutions typically rely on manually drawing masks to limit the generation range. This is not only inefficient in terms of data processing, but also only provides physical isolation in space. It fails to fundamentally solve the problem of insufficient semantic understanding of specific industrial parts by the computational model, which may result in the texture or structure in the generated area still deviating from industrial design standards.

[0072] This scheme first parses the unstructured natural language description into structured key-value pairs containing target component labels and visual attribute parameters. Using these as indexes, it automatically retrieves the corresponding domain-adaptive feature parameters from a pre-built library before the computation process begins. In subsequent iterative denoising computations, these retrieved parameter data are dynamically loaded and merged into the weight matrix of the attention layer. This data processing method allows a general diffusion computation architecture to temporarily acquire the ability to process specific industrial components within a given computational task without altering its original infrastructure. Compared to conventional methods that rely solely on text-based conditional vectors for generation, this parameter injection mechanism based on structured data enhances the semantic response accuracy to specific design objects from the computational level, ensuring that the generated components conform to specific design intents in terms of geometric structure and texture details. Furthermore, in each step of the iterative denoising computation, a region freezing computation based on component region masks is performed. Specifically, this step introduces a data stream that performs forward noise addition computation on the original image, weightedly fusing the original image noise data calculated at the current time step with the denoised data predicted by the model based on mask tensors. This mathematical constraint in the latent feature space forces the feature evolution trajectory of the non-edited region to be strictly locked within the signal distribution of the original image, while only the edited region is allowed to evolve in a directional manner according to the injected semantic parameters. Since both undergo a complete denoising iteration calculation in the same latent space coordinate system, the final decoded image exhibits mathematical consistency in light and shadow projection, texture transition, and perspective relationships, avoiding the splicing marks or background distortions commonly found in conventional local repairs.

[0073] The foregoing description of the specifications and embodiments is intended to explain the scope of protection of this invention, but does not constitute a limitation on the scope of protection of this invention. Modifications, equivalent substitutions, or other improvements to the embodiments of this invention or a portion thereof that can be obtained by those skilled in the art through logical analysis, reasoning, or limited experimentation, based on the teachings of this invention or the foregoing embodiments, in conjunction with common knowledge, general technical knowledge, and / or existing technology, should all be included within the scope of protection of this invention.

Claims

1. A method for partial design modification of footwear based on text instructions, characterized in that, Includes the following steps: Obtain design input data, which includes the original shoe image to be modified and a natural language description of the modifications to the original shoe image; Semantic parsing and entity extraction operations are performed on the natural language modification description to construct a structured modification instruction, which includes target component label data and target visual attribute parameters; Based on the target component label data in the structured modification instruction, visual region retrieval and pixel-level segmentation operations are performed on the original shoe image to generate a component region mask indicating the target modification region; According to the structured modification instructions, the domain adaptation feature parameters associated with the target component label data mapping are called from the pre-built footwear component feature library; The original shoe image is mapped to the latent space to obtain initial latent feature data, and the structured modification instructions are converted into semantic condition vectors; An iterative denoising calculation process based on a diffusion architecture is executed. In the iterative denoising calculation process, the domain adaptation feature parameters are loaded into the calculation weights, the semantic condition vector is used as a guide, and the component region mask is used to perform region freezing calculation on the noise prediction results to obtain the denoised target potential feature data. The target latent feature data is decoded and reconstructed to output a locally modified image of the target shoe.

2. The method for partial design modification of shoes based on text instructions according to claim 1, characterized in that, The step of performing semantic parsing and entity extraction operations on the natural language modification description to construct structured modification instructions includes: The intent recognition calculation is performed on the modified natural language description using natural language processing algorithms. Extract entity words from the natural language modified description as the target component label data, and extract adjectives or noun phrases from the natural language modified description as the target visual attribute parameters; The target component label data and the target visual attribute parameters are combined into a key-value pair format data packet to obtain the structured modification instruction.

3. The method for partial design modification of shoes based on text instructions according to claim 1, characterized in that, The step of performing visual region retrieval and pixel-level segmentation operations on the original shoe image based on the target component label data in the structured modification instruction, and generating a component region mask indicating the target modification region, includes: Using the target component label data as search prompts, open vocabulary target detection is performed on the original shoe image to determine the bounding box coordinates of the target component; Using the bounding box coordinates as prompt information, pixel-level segmentation is performed on the image data within the bounding box coordinate range to generate a binarized component region mask, wherein the numerical identifiers within the component region mask are the edited regions, and the numerical identifiers outside the component region mask are the non-edited regions.

4. The method for partial design modification of footwear based on text instructions according to claim 1, characterized in that, The domain adaptation feature parameters are low-rank adaptation weight matrices; before the step of executing the iterative denoising calculation process based on the diffusion architecture, the following steps are also included: The diffusion architecture employs a multimodal diffusion backbone network; The footwear component feature library stores multiple low-rank adaptive weight matrices pre-trained for different footwear components; Based on the target component label data, the corresponding low-rank adaptive weight matrix is ​​read, and the low-rank adaptive weight matrix is ​​merged into the original weights of the attention projection layer of the multimodal diffusion backbone network.

5. The method for partial design modification of footwear based on text instructions according to claim 4, characterized in that, The method further includes a step of performing spatial structure feature guidance: Perform multi-scale convolution operations on the component region mask to extract multi-scale spatial feature data; By using residual connections, the multi-scale spatial feature data is superimposed onto the corresponding hierarchical features of the multimodal diffusion backbone network to constrain the spatial structure in the iterative denoising calculation process.

6. The method for partial design modification of shoes based on text instructions according to claim 1, characterized in that, The step of performing region freezing calculation on the noise prediction result using the component region mask includes: During each step of the denoising iteration calculation, the first noisy latent feature data is predicted based on the latent features of the current step number; Perform forward noise addition calculation on the initial latent feature data to obtain second noisy latent feature data that matches the current step number; The first noisy latent feature data and the second noisy latent feature data are weighted and fused using the component region mask. The first noisy latent feature data is retained in the edit area indicated by the component region mask, and replaced with the second noisy latent feature data in the non-edit area indicated by the component region mask. The fused latent feature data is then used for the next iterative calculation.

7. The method for partial design modification of footwear based on text instructions according to claim 1, characterized in that, The method for constructing the feature library of footwear components includes: Perform batch image generation to obtain composite image data containing footwear products and corresponding composite line art data; Based on attention map extraction algorithms or manual annotation operations, the mask annotation data of each footwear component in the synthetic image data is determined, and a triplet training dataset containing text descriptions, synthetic line drawings, and component masks is constructed. The basic diffusion model is fine-tuned using the triplet training dataset to obtain feature parameters that can reconstruct specific footwear components, and these feature parameters are stored in the footwear component feature library.

8. The method for partial design modification of footwear based on text instructions according to claim 4, characterized in that, The multimodal diffusion backbone network comprises multiple computational layers; The low-rank adaptive weight matrix is ​​only merged into specific intermediate layer intervals responsible for semantic and structural combination in the multimodal diffusion backbone network, and the intermediate layer intervals are layers 15 to 44.

9. The method for partial design modification of footwear based on text instructions according to claim 1, characterized in that, The method further includes: The target shoe image is displayed through a display interface; When a further modification instruction is received for the target shoe image, the target shoe image is defined as a new original shoe image, and the parsing, retrieval, segmentation, mapping, and iterative denoising calculation process is repeated.

10. A text-based instruction-based system for modifying partial designs of footwear, characterized in that, include: The input unit is used to acquire design input data, which includes the original shoe image to be modified and a natural language modification description of the original shoe image. The instruction parsing unit is used to perform semantic parsing and entity extraction operations on the natural language modification description to construct a structured modification instruction, which includes target component label data and target visual attribute parameters. The visual computing unit is used to perform visual region retrieval and pixel-level segmentation operations on the original shoe image according to the target component label data in the structured modification instruction, and generate a component region mask indicating the target modification region. The resource scheduling unit is used to call the domain adaptation feature parameters associated with the target component label data mapping from the pre-built footwear component feature library according to the structured modification instructions. The denoising generation unit is used to map the original shoe image to the latent space to obtain initial latent feature data, and to convert the structured modification instructions into semantic condition vectors; to execute an iterative denoising calculation process based on a diffusion architecture, in which the neighborhood adaptation feature parameters are loaded into the calculation weights, the semantic condition vector is used as a guide, and the component region mask is used to perform region freezing calculation on the noise prediction results; finally, the calculated target latent feature data is decoded and reconstructed to output the locally modified target shoe image.

Images