Shoe style fusion generation method and system based on text and image

Abstract

The invention provides a text and image-based shoe style fusion generation method and system. The method comprises the following steps of: firstly, acquiring design input data comprising a reference image and a natural language instruction; performing semantic analysis on the instruction, and determining a target component, a source index and a mixed weight; then, constructing an element condition sequence, and performing coding combination on the visual features of the source reference image, the text semantic features of the target component and the mixed weight to obtain element feature vectors; and inputting the sequence into a generative model, and establishing semantic response mapping between element feature vectors and a generated image space region by using a cross attention mechanism, thereby accurately controlling visual performance of each target component without a mask, and finally generating and outputting a fusion design drawing. According to the method, shoe style fusion generation is performed by using the large language model, and the design efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of shoe design, and particularly relates to a shoe fusion generation method and system based on text and images. BACKGROUND

[0002] In the field of modern shoe industry design, designers often need to extract specific parts from multiple existing shoe samples for reorganization, such as combining the sole structure of a classic sports shoe with the upper material of another casual shoe to quickly explore new design solutions. Current shoe fusion design still mainly relies on manual operation by designers, and the design efficiency is low. SUMMARY

[0003] The present application aims to overcome the above-mentioned defects or problems in the background art, and provides a shoe fusion generation method and system based on text and images, which uses a large language model to generate shoe fusion, and can effectively improve design efficiency.

[0004] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: Technical solution one: a shoe fusion generation method based on text and images, comprising the following steps: obtaining design input data of a user, the design input data containing at least one reference image and natural language instructions; performing semantic analysis on the natural language instructions to determine target parts to be generated, source reference image indexes corresponding to each target part, and mixing weights; constructing an element condition sequence according to the design input data, wherein for each target part, the visual features of the source reference image corresponding to the target part are extracted, and the visual features are combined with the text semantic features of the target part and the mixing weights to obtain an element feature vector in the element condition sequence; inputting the element condition sequence into a generation model, establishing semantic response mapping between the element feature vector and the spatial region of the generated image by using a cross-attention mechanism, and controlling the visual performance of each target part in the generated image according to the semantic response mapping; obtaining at least one candidate fusion image output by the generation model, and outputting a final design image.

[0005] Technical solution two based on technical solution one: the semantic analysis on the natural language instructions comprises: identifying entity words in the natural language instructions by using a natural language processing model, mapping the entity words to categories in a preset standard part set, and the standard part set contains shoe part categories; when the natural language instructions specify multiple source reference images for the same target part, the initial weight values corresponding to each source reference image are analyzed respectively.

[0006] The step of extracting the visual features of the corresponding source reference image comprises: extracting high-dimensional hidden layer features of the source reference image through an image encoding network; and extracting a color palette style vector of the source reference image through a color extraction algorithm or a style encoder; and the element feature vector is formed by splicing a text embedding vector of the target component, the high-dimensional hidden layer features of the source reference image, the mixing weight, and the color palette style vector.

[0007] The step of constructing the element condition sequence according to the design input data further comprises: performing dimension projection on the spliced element feature vector by using a multi-layer perception machine, so that the dimension of the element feature vector is consistent with the dimension of a key-value pair of an internal cross-attention layer of the generative model.

[0008] The processing procedure of the cross-attention mechanism when the element condition sequence is input into the pre-trained generative model comprises: when a plurality of source reference images correspond to the same target component, performing temperature adjustment processing on the mixing weight corresponding to the plurality of source reference images by using a normalized exponential function, to obtain a normalized weight; and performing weighted aggregation on the visual features from different source reference images according to the normalized weight, to generate aggregated component control features.

[0009] The step of establishing semantic response mapping between the element feature vector and a spatial region of a generated image by using a cross-attention mechanism comprises: activating a spatial attention map inside the generative model by using the text semantic features in the element feature vector, to determine a distribution region of the target component in the generated image; and using the spatial attention map as a spatial distribution constraint, to guide the visual features in the element feature vector to be injected into the distribution region, to complete local feature migration.

[0010] The step of outputting a final design drawing comprises: controlling the generative model to generate a plurality of candidate fusion images; inputting the candidate fusion images into a preset multi-dimensional scoring module, to calculate scores of the candidate fusion images in the dimensions of structural rationality, style consistency, color coordination, and material authenticity; sorting the candidate fusion images according to the scores, and selecting an image with the highest score as the final design drawing output.

[0011] Technical solution eight based on technical solution seven: The multi-dimensional scoring module executes conflict detection logic, including: calculating the cosine similarity between the feature vector of the candidate fused image and the feature vector of the weighted reference image; and reducing the score of the candidate fused image when the cosine similarity is lower than a preset threshold or the structural proportion parameter of the candidate fused image deviates from a preset proportion range.

[0012] Technical solution nine based on technical solution seven: the output final design drawing further includes: generating and outputting text optimization suggestions for the final design drawing based on the score of the multi-dimensional scoring module; the text optimization suggestions include recommendations for adjusting the hybrid weights or material parameters.

[0013] Furthermore, this invention also provides technical solution ten: a footwear fusion generation system based on text and images, comprising: an input acquisition module for receiving multiple reference images and natural language instructions; a parsing and processing module for performing semantic parsing on the natural language instructions, determining the target component, source reference image index, and mixing weights, and extracting the visual features of the reference images; a sequence construction module for constructing an element condition sequence based on the parsing results, the element condition sequence containing element feature vectors that fuse text semantic features, visual features, and mixing weights; a fusion generation module for inputting the element condition sequence into a generation model, establishing a semantic response mapping between the visual features and the spatial regions of the generated image through a cross-attention mechanism to generate candidate fused images; and a preferred output module for evaluating and ranking the candidate fused images and outputting the final design drawing.

[0014] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects: Technical Solution 1 simplifies the shoe fusion design process by introducing a large language model. Designers only need to provide natural language instructions with design intent and reference images for fusion design, and the large language model can automatically generate and output the fused shoe design, thereby improving the efficiency of shoe fusion design. Furthermore, this solution achieves accurate recognition and automated shoe fusion without relying on manually set masks. In existing image fusion or editing technologies, to migrate specific components (e.g., the sole) from the first reference image to the corresponding position in the second reference image, the common practice is to rely on explicit spatial constraints, requiring users to manually draw high-precision binary masks or pre-generate segmentation maps using semantic segmentation networks. However, footwear products typically contain complex pattern structures and irregular edge features, such as interwoven eyelets, midsoles with gradient textures, or porous mesh uppers. These areas often lack clear binarization boundaries at the pixel level. Conventional masking techniques, if the edge delineation is not precise enough, or if the segmentation network deviates from recognizing sketches from non-standard perspectives, will result in jagged edges, artifacts, or structural breaks in the fused image at the joints of components, failing to meet the requirements of industrial design for realism and structural continuity.

[0015] This scheme first encodes and combines the textual semantic features of the target component with the visual features of the source reference image and hybrid weights during the data preprocessing stage to construct a tightly coupled element feature vector. Then, during the inference process of the generative model, a cross-attention mechanism is used to parse the aforementioned element feature vector. Since the cross-attention mechanism calculates the correlation between the input conditions and the intermediate features of the generated image, the textual semantic components in the element feature vector can activate the corresponding spatial attention map within the generative model. This allows for the automatic determination of the probability distribution region of the target component on the generated image plane based on semantic understanding; for example, the semantic information of "midsole" automatically leads to a high response in the lower part of the image. This spatial attention map, activated by textual semantics, constitutes an implicit, soft spatial distribution constraint. This application utilizes this constraint to guide the visual features in the element feature vector to be injected only into the high-response spatial region, thereby achieving automatic alignment of visual features and semantic regions without the need for explicit input coordinates or masks. Unlike conventional hard mask clipping, the response mapping generated by the cross-attention mechanism is a continuous probability distribution. This allows component features from different sources to smoothly transition at the boundaries according to the statistical laws of image generation, preserving the natural gaps and light and shadow transitions of the footwear product. Meanwhile, by directly introducing hybrid weights when constructing the element conditional sequence, the fusion of multi-source features occurs in the hidden feature space rather than the pixel space. The model can complete the mathematical aggregation of multi-dimensional features before feature decoding based on the weights, thus presenting a fusion effect with a rigorous structure and natural texture in the final output design.

[0016] In technical solution two, by establishing a mapping relationship between entity words in natural language instructions and a preset set of standard components, semantic ambiguity in non-standardized user descriptions is eliminated, ensuring that the generated model can accurately correspond to the specific structure of footwear. Simultaneously, when the same component involves multiple sources, the initial weight values ​​are parsed separately, enabling the model to quantify and distinguish the feature contributions of different sources in subsequent processing. This achieves precise proportional control during the feature mixing stage, avoiding confusion between primary and secondary features due to unclear weight allocation.

[0017] In technical solution three, high-dimensional hidden layer features and color palette style vectors of the source reference image are extracted respectively, and they are concatenated with text embedding vectors and mixing weights. This enables the generation model to independently acquire and utilize texture details and color distribution, preventing color deviation or texture blurring caused by feature entanglement during the fusion process. This ensures that the generated parts can accurately reproduce the expected color style while maintaining the material texture of the reference image.

[0018] In technical solution four, a multilayer perceptron is used to project the spliced ​​element feature vectors into dimensions, mapping the composite vector containing multidimensional information to the key-value pair dimension that the model's cross-attention layer can accept. This ensures that externally injected control signals can participate in the model's attention calculation, guaranteeing the smooth flow of feature injection channels and the numerical stability of the model's inference process.

[0019] In technical solution five, a normalized exponential function temperature adjustment process is introduced when processing multi-source feature fusion. This allows the original weights to be smoothed or sharpened according to the temperature parameter, thereby adjusting the blending smoothness of features from different sources during the feature aggregation stage. This avoids the feature from a certain source being over-amplified or suppressed due to the linear difference in the original weight values, so that the final generated component can more naturally present the expected blending effect of features from multiple reference images.

[0020] In technical solution six, the spatial attention map activated by text semantic features is used as a clear spatial distribution constraint, which forces visual features to take effect only in the pixel area corresponding to the semantic logic. This effectively suppresses the accidental overflow of texture features to the background area or adjacent component area, thereby significantly improving the boundary clarity and positional accuracy of local feature migration without human intervention.

[0021] Technical Solution 7 integrates a multi-dimensional scoring module that includes four dimensions: structural rationality, style consistency, color coordination, and material authenticity. This allows for the automatic identification and elimination of inferior images with structural distortions, stylistic inconsistencies, or material distortions from the randomly distributed results output by the generated model. This ensures that the final output design drawings meet industrial design standards in both aesthetics and physical logic, reducing the time cost for users to select effective solutions from a large number of candidate images.

[0022] In technical solution eight, a conflict detection logic based on cosine similarity and structural proportion parameters is executed in the scoring module. This automatically detects illusory images that, although clear, deviate significantly from the features of the reference image or violate the basic structural proportions of footwear, ensuring that the fused design drawings are feasible for industrial applications.

[0023] In technical solution nine, targeted text optimization suggestions are generated based on the scoring results. By analyzing the reasons for low scores, specific adjustment directions are provided to users, helping them quickly locate the root cause of poor generation results. This can shorten the design cycle of shoe integration and thus improve design efficiency.

[0024] Technical Solution 10 provides a text- and image-based shoe fusion generation system. This system integrates functional modules such as input acquisition, parsing and processing, sequence construction, fusion generation, and optimized output. The sequence construction module integrates discrete natural language instructions and reference image features into a structured element condition sequence at the data level. In conjunction with the cross-attention mechanism in the fusion generation module, the system automatically establishes a spatial mapping relationship between semantic and visual features, thereby replacing the manual operation workflow in traditional design methods. This enables direct routing and fusion of multi-source shoe components in the feature space. At the same time, the optimized output module achieves automated quality control of the generated results, effectively improving the efficiency of shoe fusion design. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments are briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the method for generating shoe designs based on text and images according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the modules of the shoe fusion generation system based on text and images involved in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are preferred embodiments of the present invention and should not be considered as excluding other embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] The illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0029] Unless otherwise expressly defined, the use of terms such as "first," "second," or "third" in the claims, description, and accompanying drawings of this invention is for distinguishing different objects and not for describing a specific order.

[0030] In the claims, description and accompanying drawings of this invention, the terms "comprising," "having," and variations thereof are used to mean "including but not limited to."

[0031] Reference Figure 1 The diagram illustrates a flowchart of a method for generating shoe designs based on text and images, according to an embodiment of the present invention. Also, referring to... Figure 2 The diagram illustrates a module schematic of a text and image-based shoe fusion generation system according to an embodiment of the present invention.

[0032] The system mainly includes the following modules: The input acquisition module is used to receive multiple reference images and natural language commands; The parsing and processing module is used to perform semantic parsing on the natural language instructions, determine the target component, the source reference image index and the mixing weight, and extract the visual features of the reference image; The sequence construction module is used to construct an element condition sequence based on the parsing results. The element condition sequence includes an element feature vector that integrates text semantic features, visual features, and mixed weights. The fusion generation module is used to input the element condition sequence into the generation model and establish a semantic response mapping between the visual features and the spatial regions of the generated image through a cross-attention mechanism to generate candidate fused images. The preferred output module is used to evaluate and sort the candidate fused images and output the final design drawing.

[0033] This system is used to implement the aforementioned method. The specific steps of the method will be described in detail below, focusing on the specific modules of the system. This detailed explanation will enable those skilled in the art to clearly understand the specific structure of the system. Furthermore, after understanding the method, those skilled in the art will be able to implement the system based on common knowledge in the field.

[0034] like Figure 1 As shown, the method mainly includes the following steps: S110, Obtain the user's design input data, which includes at least one reference image and natural language instructions.

[0035] This step is performed through an input acquisition module. For example, multimodal input data submitted by users for a specific shoe design task can be acquired through a human-computer interaction interface (HCI). The HCI can be deployed on mobile terminals, PCs, or web pages, providing image upload controls and text input boxes. The design input data mainly includes two parts: The first part consists of reference images. Users can upload images from local storage devices or select one or more images from the system's pre-built shoe pattern database as baseline materials for their designs. These reference images can be high-resolution photographs, 3D renderings, hand-drawn sketches, or style samples. Optionally, after receiving the reference images, the system will perform format verification and standardization preprocessing on the image files. For example, images in different formats (such as JPG, PNG, WEBP, etc.) will be converted to a unified tensor format, and the resolution will be adjusted to a standard size (such as 1024×1024 pixels) suitable for the generated model. Simultaneously, normalization processing will be performed to eliminate differences in lighting or color space, ensuring the stability of subsequent feature extraction.

[0036] The second part consists of natural language instructions. Users describe their design intent using natural language, particularly regarding the selection or omission of specific components from different reference images, material replacements, or style integration methods. Optionally, these instructions support text input or speech-to-text input and allow the inclusion of vague modifiers (such as "more sporty" or "retain the retro feel") or explicit structural specifications (such as "use an A-shaped base").

[0037] To more clearly illustrate the workflow of this solution, the following example demonstrates a "retro-technology fusion" shoe design: In this embodiment, the user wants to design a new type of athletic shoe that combines classic vulcanized shoe aesthetics with modern cushioning technology. The user uploaded two reference images through the interface: Reference Image A (Ref_A): A classic high-top canvas shoe image, characterized by a beige canvas upper, a prominent circular logo on the side, and a vulcanized rubber outsole; Reference Image B (Ref_B): A modern functional running shoe image, characterized by neon green breathable mesh material, a complex TPU support structure, and a thickened air-cushioned midsole. Meanwhile, the user's natural language input command is: "Use the upper structure of Image A, retain the logo of Image A; use the air-cushioned midsole of Image B; replace the upper material with the mesh of Image B; the color scheme should blend the styles of both." After receiving Image A, Image B, and the text command, the system packages them into design input data and passes it to the subsequent processing module.

[0038] S120, perform semantic parsing on the natural language instructions to determine the target component to be generated, the source reference image index corresponding to each target component, and the hybrid weight.

[0039] After receiving the natural language instructions, the system processes them using a built-in parsing module. In this embodiment, this parsing module is built on a large language model (LLM, such as the Llama series or the Qwen model fine-tuned for the design domain). The model is trained to understand industrial design terminology and can extract structured control parameters from unstructured natural language text. Specifically, the parsing process aims to output a structured parsing list (e.g., in JSON format) that defines "which image each component in the generated image should originate from" and "what the retention level (weight) should be."

[0040] S120 further includes the following sub-steps: S121, using a natural language processing model to identify entity words in the natural language instruction, and mapping the entity words to categories in a preset standard component set, the standard component set including footwear component categories; In practical applications, users often use inconsistent terminology for footwear components (e.g., calling the "outsole" the "sole plate" and the "upper" the "leather upper"). To ensure precise control of the subsequent generated model, step S121 performs entity standardization mapping. Specifically, the system pre-defines a set of standard components that covers all key areas of footwear design, such as: Upper_All (overall upper), ToeCap (toe cap), Vamp (forefoot), Quarter (shoe body), HeelCounter (heel counter), Tongue (tongue), Laces (laces), Midsole (midsole), Outsole (outsole), Logo (brand logo), etc. The natural language processing model first segments and identifies entities in the instruction text, extracting the entity words mentioned by the user; then, through semantic similarity calculation or lookup table method, these entity words are mapped to unique category IDs in the aforementioned set of standard components.

[0041] In the aforementioned case of "fusion of retro canvas shoes and modern running shoes," the instruction text includes: "Upper Structure" identifies the entity "Upper" and maps it to the standard category Upper_Structure; "Logo" identifies the entity "Logo" and maps it to the standard category Logo; "Cushion Midsole" identifies the entity "Midsole" and maps it to the standard category Midsole; "Upper Material" identifies the entity "Material" and maps it to the standard category Material_Texture.

[0042] S122, when the natural language instruction specifies multiple source reference images for the same target component, the initial weight value corresponding to each source reference image is parsed respectively; When a user's instruction involves multi-source fusion (e.g., "fusion of color schemes" or "balancing the outline of A and the details of B"), the contribution ratio of different sources needs to be clearly defined. Specifically, the parsing module analyzes the degree adverbs or numerical descriptions in the instruction: if the instruction explicitly specifies the ratio (e.g., "mainly refer to Figure A, with a slight reference to Figure B"), the model will parse out the corresponding weight values; if the instruction does not explicitly specify the ratio (e.g., "fusion of styles"), the system will default to assigning equal weights; if the instruction specifies "use Figure A's...", it implies that the weight of Figure A corresponding to that component is 1.0, and the weight of Figure B is 0.0.

[0043] Based on the aforementioned input instructions ("Use the upper structure of Figure A, retain the logo of Figure A; use the air cushion midsole of Figure B; replace the upper material with the mesh of Figure B; the color scheme blends the styles of both"), after processing in steps S120 to S122, the structured parsing result (JSON example) generated by the system is shown below: { "fusion_plan": [ { "target_part": "Upper_Structure", / / Target part: Upper structure "sources": [ { "img_index": "Ref_A", "weight": 1.0} / / Source: Image A, weight 1.0 (completely inherits the outline of A) ]}, { "target_part": "Logo", / / Target part: Logo "sources":[ { "img_index": "Ref_A", "weight": 1.0} / / Source: Image A, weight 1.0 ]}, { "target_part": "Midsole", / / Target part: Midsole "sources": [ { "img_index": "Ref_B", "weight": 1.0} / / Source: Image B, weight 1.0 (uses the air cushion of B) ]}, { "target_part": "Material_Texture", / / Target part: Material texture "sources": [ { "img_index": "Ref_B", "weight": 1.0} / / Source: Image B, weight 1.0 (using mesh size B) ]}, { "target_part": "Color_Palette", / / Target part: global color scheme "sources": [ { "img_index": "Ref_A", "weight": 0.5}, / / Source: Image A (off-white) { "img_index": "Ref_B", "weight": 0.5} / / Source: Image B (neon green), a mix of both]} ]} S130, construct an element condition sequence based on the design input data, wherein, for each target component, extract the visual features of the corresponding source reference image, and encode and combine the visual features with the text semantic features of the target component and the hybrid weight to obtain the element feature vector in the element condition sequence.

[0044] This step is performed by the system's sequence construction module. This module receives the structured parsing results output from step S120 and transforms the abstract fusion plan into a tensor sequence that can be directly processed by the generative model (such as a Transformer-based diffusion model).

[0045] S130 further includes the following sub-steps: S131, extract the high-dimensional hidden layer features of the source reference image through an image coding network; extract the color palette style vector of the source reference image through a color extraction algorithm or a style encoder; In the sequence construction module, the system first performs feature decoupling extraction on the reference images involved in the design input data. On one hand, a pre-trained image encoding network is used to process the reference images to obtain high-dimensional hidden layer features. In this embodiment, the image encoding network adopts CLIP Vision Model or VAE Encoder. When the system processes reference image B (modern running shoes), the features extracted by the encoding network not only include the pixel information of the image, but also high-level semantic visual information such as "mesh texture", "air cushion transparency" and "streamlined contour". Similarly, for reference image A (retro canvas shoes), the extracted features focus on information such as "canvas texture" and "vulcanized rubber texture". On the other hand, in order to achieve precise color control, the module uses color extraction algorithms (such as K-means clustering) or style encoders to extract color palette style vectors of the reference images. For example, a beige main color vector representing the retro style is extracted from image A, and a neon green main color vector representing the technological feel is extracted from image B. These vectors will serve as key parameters for controlling the color tone of the generated images in the future.

[0046] S132, the element feature vector is formed by concatenating the text embedding vector of the target component, the high-dimensional hidden layer features of the source reference image, the mixing weights, and the color palette style vector; The sequence construction module assembles an initial composite element feature vector for each "target component" in the parsed list and its corresponding "source reference image". To ensure that multimodal information can be uniformly processed by the model, the construction of this vector strictly follows the mathematical logic of feature concatenation. Specifically, for the i-th target component and its corresponding j-th source reference image, its original element feature vector... The construction formula is as follows:

[0047] In the above formula, the symbol This represents a vector concatenation operation. The parameters are defined as follows: This represents the text semantic embedding vector obtained after the name of the i-th target component (e.g., "Midsole") has been processed by the text encoder. This represents the high-dimensional hidden layer features extracted from the j-th source reference image through an image coding network; Represents the user-specified mixed weight values ​​(or their vectorized representation); This represents the palette style vector extracted from the source reference image. The formula explicitly defines the smallest control unit for the model input, namely a full-feature tensor containing "semantic-visual-weight-style".

[0048] S133, The spliced ​​element feature vector is dimensionally projected using a multilayer perceptron, so that the dimension of the element feature vector is consistent with the dimension of the key-value pairs in the cross-attention layer inside the generation model. Since the vectors obtained by direct concatenation are usually of variable length and extremely high dimensionality—for example, the direct concatenation of text embeddings, image features, weight values, and style vectors can result in dimensions reaching thousands of dimensions, making them unsuitable for direct input into standard generative models—the sequence construction module includes a learnable multilayer perceptron (MLP) projection layer. This projection layer maps the concatenated long vector to the key / value pair dimension (e.g., 768 or 1024 dimensions) required by the cross-attention layer within the generative model (such as a Transformer-based diffusion model). After projection, the final element-conditional sequence is shown in the pseudocode structure below, and this sequence will be fed as the sole conditional input into the subsequent fusion generation module.

[0049] The pseudocode is shown below: Element_Condition_Sequence = [ Vector_Upper(Text="Upper", Visual=Ref_A, Weight=1.0, Style=Ref_A), Vector_Logo(Text="Logo", Visual=Ref_A, Weight=1.0, Style=Ref_A), Vector_Midsole(Text="Midsole", Visual=Ref_B, Weight=1.0, Style=Ref_B), Vector_Material(Text="Texture", Visual=Ref_B, Weight=1.0, Style=Ref_B) ] S140, the element conditional sequence is input into the generation model, a semantic response mapping between the element feature vector and the spatial region of the generated image is established using a cross-attention mechanism, and the visual representation of each target component in the generated image is controlled according to the semantic response mapping.

[0050] This step is performed by the system's fusion generation module. At its core is a finely tuned conditional diffusion model (e.g., a latent diffusion model based on U-Net or Transformer architecture). The module receives the element conditional sequence from the sequence construction module as Key and Value, and the latent variables of the noisy image during the generation process as Query. At each denoising time step in the generation process, the model uses a cross-attention mechanism to calculate the correlation between each spatial pixel of the noisy image and the tokens of each component in the element conditional sequence. This correlation, known as the "semantic response mapping," acts as an implicit guidance signal, telling the model where to draw the features of which component on the canvas, thus achieving automated and precise alignment of structure and texture without requiring the user to manually draw complex masks.

[0051] S140 further includes the following sub-steps: S141, using the textual semantic features in the element feature vector to activate the spatial attention map inside the generation model, and determining the distribution area of ​​the target component in the generated image; During the cross-attention calculation process, the fusion generation module utilizes the embedded textual semantic features (such as "Midsole / midsole") in the element feature vectors to activate the spatial attention map within the model. Since the pre-trained generative model already possesses prior knowledge of the shoe structure, when the model receives the textual semantic "Midsole," its internal attention mechanism automatically generates a high response value (i.e., a highlighted heatmap) in the bottom region of the generated image, and a low response value in the top region. Taking the "retro and technological fusion" design in this embodiment as an example, when processing a feature vector containing the semantic "Midsole," the generated spatial attention map automatically highlights the sole area; when processing a feature vector containing the semantic "Logo," the attention map automatically focuses on the circular area on the side of the shoe upper. This mechanism effectively utilizes semantic information to automatically generate a soft attention mask.

[0052] S142, using the spatial attention map as a spatial distribution constraint, guides the visual features in the element feature vector to be injected into the distribution area, thereby completing local feature transfer; The fusion generation module uses the generated spatial attention map as a spatial distribution constraint. During the feature injection stage, based on the weight distribution of the attention map, the system injects the visual features of the source reference image carried in the element feature vector (such as the air cushion texture in Figure B) only into the high-response distribution region. This means that although the input is a complete reference image feature set, only the air cushion texture corresponding to the "midsole" position is routed to the bottom of the generated image, while other irrelevant information in the reference image (such as the background and shoelaces) is automatically ignored. In this way, the system achieves the grafting of the air cushion midsole in Figure B onto the lower part of the shoe body in Figure A without disrupting the overall structural coherence of the image. Furthermore, the transition at the junction is naturally smoothed by the denoising process of the diffusion model, avoiding the jagged edges produced by traditional hard splicing.

[0053] S143, when the element condition sequence is input into the pre-trained generative model, when the same target component corresponds to multiple source reference images, the mixed weights corresponding to the multiple source reference images are subjected to normalized exponential function temperature adjustment processing to obtain normalized weights. When design instructions involve multi-source fusion (e.g., "color scheme fusion of image A and image B"), the fusion generation module needs to process multiple feature vectors corresponding to the same target component. To precisely control the contribution of different sources, the module introduces a normalized exponential function with a temperature coefficient to process the user-defined initial weights. Specifically, for the first... The component comes from the first The original weights of each source Its normalized weight The calculation formula is as follows:

[0054] In the formula, This is a learnable or preset temperature parameter. When... When the value is large, the weight distribution tends to be sharp, and the model tends to retain only the single source feature with the highest weight; when When the values ​​are small, the weight distribution tends to be smooth, and the model tends to evenly mix features from multiple sources. This mechanism ensures that in multi-source fusion scenarios, the mixing ratio of features strictly follows the user's design intent, while avoiding generation crashes caused by numerical instability.

[0055] S144, according to the normalized weights, the visual features from different source reference images are weighted and aggregated to generate aggregated component control features; After obtaining the normalized weights, the fusion generation module performs a weighted aggregation operation in the feature space to generate the final aggregated features used to guide denoising. The polymerization process is illustrated by the following formula:

[0056] In the formula, This represents the aforementioned dimensional projection function (such as an MLP layer). Through this step, the off-white color scheme features from Figure A and the neon green color scheme features from Figure B are mathematically fused in the latent vector space, generating a new hybrid feature that combines retro tones with a technological fluorescent feel. This aggregated feature then participates in the inverse denoising process of the diffusion model, ultimately rendering a visually harmonious and unified fused shoe image.

[0057] S150, acquire at least one candidate fusion image output by the generative model, and output the final design drawing.

[0058] This step is performed by the system's optimal output module. While generative models possess powerful creativity, the generation process based on diffusion models is random, potentially resulting in structurally flawed or physically incompatible images. Therefore, this system does not directly output a single result. Instead, it first acquires a set of candidate images and filters them through an automated evaluation mechanism to ensure that the final design delivered to the user is aesthetically and industrially viable.

[0059] S150 further includes the following sub-steps: S151, control the generative model to generate multiple candidate fused images; During the inference phase, the preferred output module sends instructions to the generative model, requiring it to generate multiple (e.g., K=4) candidate fused images in parallel by changing the initial random noise seed under the same input condition sequence. In the "Retro and Technological Fusion" case of this embodiment, the system generates four candidate images: Figure 1 It features a retro upper combined with an air-cushioned sole; candidate Figure 2 Although both colors were combined, the sole showed signs of cracking; candidate image 3 omitted the neon green color scheme, retaining only the off-white; and candidate image 4 exhibited excessively distorted textures. These images were then submitted to the scoring process.

[0060] S152, input the candidate fused images into a preset multi-dimensional scoring module to calculate the scores of each candidate fused image in the dimensions of structural rationality, style consistency, color coordination and material authenticity; The system incorporates a pre-trained multi-dimensional scoring module, typically based on ResNet or ViT architectures and fine-tuned on large-scale footwear datasets. The module contains four independent regression prediction heads, each scoring a candidate image (0-100 points). For the four candidate images, the module performs the following evaluations: Structural rationality checks the completeness of the shoe outline and the natural connection between the midsole and upper; Style consistency checks whether it simultaneously embodies a retro and technological feel; Color harmony checks whether the generated colors match the input color palette vector; and Material realism checks the realism of the mesh and canvas texture rendering.

[0061] S153, execute conflict detection logic: calculate the cosine similarity between the feature vector of the candidate fused image and the feature vector of the weighted reference image; if the cosine similarity is lower than a preset threshold, or if the structural proportion parameter of the candidate fused image deviates from the preset proportion range, reduce the score of the candidate fused image; To prevent illusions, the preferred output module performs rigorous conflict detection. First, the system extracts high-dimensional feature vectors from the candidate fused image and simultaneously calculates the weighted average of the feature vectors from all source reference images. Then, it calculates the cosine similarity between these two vectors. If this similarity value is below a preset threshold (e.g., 0.75), it indicates a significant discrepancy between the generated image and the reference material, and the system will drastically reduce its overall score. Simultaneously, the module uses image processing algorithms to extract the bounding rectangle of the generated image and calculates its aspect ratio and other structural proportion parameters. If these parameters significantly deviate from the preset proportion range of a standard athletic shoe (e.g., candidate image 4 is identified as having an abnormal aspect ratio, resembling a sock rather than a shoe), the system will also penalize it with a deduction of points.

[0062] S154, Sort the candidate fusion images according to the scores, and select the image with the highest score as the final design image for output; The system calculates a weighted total score for each candidate image based on the scores across the four dimensions mentioned above and the penalty results from conflict detection. In this embodiment, the candidate... Figure 1 Structurally, it fully retains the essence of vulcanized shoes and the shape of the air cushion; color-wise, it perfectly blends off-white and neon green, and does not trigger any conflict penalties, thus obtaining the highest score (e.g., 92 points). The optimal output module then marks it as the Top-1 result and renders it to the user through the UI interface as the final suggested design solution.

[0063] S155, Based on the score from the multi-dimensional scoring module, generate and output text optimization suggestions for the final design drawing; the text optimization suggestions include recommendations for adjusting the mixed weights or material parameters.

[0064] While outputting the image, the system uses feedback from the scoring module to generate intelligent text suggestions. If the final selected image scores slightly lower in "Material Realism" than other items (e.g., the boundary between canvas and mesh appears slightly harsh), the system will call the preset rule engine or language model to generate the following optimization suggestions and display them in the sidebar: "The current mesh material coverage weight (1.0) may cause texture conflict with the canvas logo area. It is recommended to lower the 'Material' weight to 0.8 or enable the 'Edge Softening' parameter to obtain a more harmonious visual effect." This not only delivers the design result but also assists the user in the next round of iterative optimization.

[0065] To enable the generative model of this invention to possess precise local control and multi-source fusion capabilities, in one embodiment of this application, training data is pre-constructed using a bootstrapping strategy, and the model is trained using a composite loss function. The specific process is as follows: Training data construction: This invention employs a dual-path data synthesis strategy to address the lack of high-quality paired training data in a specific domain (footwear fusion).

[0066] Forward Fitting Chain: This strategy simulates the iterative process of a designer "from sketch to finished product." The system first retrieves images from the database that match the real target image. Similar base maps (For example, a basic white shoe).

[0067] Step 1: Generate the instruction "Modify the sole as the target" The style utilizes existing strong editing models (such as Qwen-Image-Edit) to generate intermediate images. .

[0068] Step Two: Based on The command to generate "modify the upper material to the target" is used. "Weaving texture", generating intermediate image .

[0069] Step 3: Based on The command "Adjust color scheme to navy blue" generates a near-target image. The final image .

[0070] Output: Through the above process, multiple sets of gradually changing training sample pairs and continuous evolutionary chains are generated, for example... and This allows the model to learn how to execute complex fusion instructions step by step.

[0071] Inverse perturbation chain: This strategy aims to improve the robustness and resilience of the model. The system starts with real, high-quality footwear images and applies appropriate parametric offsets to them.

[0072] Operations: Perform hue rotation, geometric micro-deformation, or texture noise injection on real images.

[0073] Training objective: To construct instruction pairs that "restore to reality". For example, the input is "color-distorted shoes", the instruction is "restore to standard off-white", and the goal is "original realistic image". This allows the model to learn what the "correct" physical properties are.

[0074] Generative model training: This invention uses a pre-trained diffusion editing model (such as Qwen-Image-Edit) as a base to perform full parameter fine-tuning or LoRA fine-tuning.

[0075] Input Construction: The input construction during training is consistent with that during the inference phase (S130). That is, the component words (such as "mid-bottom"), source map features, weights, and style vectors in the training samples are encapsulated into element-conditional sequences.

[0076] Loss function framework: A composite loss function is used during the training process. The optimized formula is as follows:

[0077] The definitions and effects of each type of loss are as follows: (Denoising master loss): Based on the mean square error, the difference between the noise predicted by the model and the actual added noise is calculated. This is the core of the diffusion model's ability to learn image generation.

[0078] (Image-text consistency): Constrain the consistency between the generated image features and the input text instruction features in the semantic space to ensure that "what is drawn is what is written".

[0079] (Perceptual Reconstruction Loss): The reconstruction error is calculated in the VAE latent space or feature space to ensure that the overall structure of the shoe (such as contour closure and perspective relationship) does not collapse.

[0080] (Semantic Preservation / Content Consistency Loss): By introducing a keep-tokens gating mechanism, for areas not mentioned for modification in the instructions (such as "Keep Logo"), the feature differences between the generated image and the original image in the corresponding areas are calculated, forcing the model to keep the original in non-edited areas.

[0081] (Preference Optimization Loss): Based on the DPO (Direct Preference Optimization) algorithm, the model probability distribution is adjusted using the preference pair data generated by the following ratingr, making it more inclined to generate high-scoring images that conform to human aesthetics.

[0082] Scorer training: To achieve optimal output during the inference phase, this invention pre-trains a multi-head discrimination model.

[0083] Negative example construction: The system automatically constructs "error" samples, such as: forcibly splicing two physically mutually exclusive material textures (material conflict), forcibly mixing warm and cool colors (color mismatch), or randomly shuffling the source index (source error).

[0084] Multi-head training: The discriminator model shares the same visual feature extraction backbone network, with four independent fully connected layers at the ends, outputting: structural score (for determining if the shoe body is complete); style score (for determining if the style is consistent); color score (for determining if the color scheme is harmonious); and material score (for determining if the texture is realistic). This discriminator is used not only for image selection during the inference phase but also for generating preference data during the training phase to guide the DPO training of the generative model.

[0085] The present invention relates to a text- and image-based shoe fusion generation method and system. By introducing a large language model, it simplifies the shoe fusion design process. Designers only need to provide natural language instructions with design intent and reference images for fusion design, and the large language model can automatically generate and output shoe fusion design drawings, thereby improving the efficiency of shoe fusion design. Furthermore, this solution achieves accurate recognition and automated shoe fusion without relying on manually set masks. In existing image fusion or editing technologies, to migrate specific components (e.g., soles) from a first reference image to a corresponding position in a second reference image, the common practice is to rely on explicit spatial constraints, requiring users to manually draw high-precision binary masks or pre-generate segmentation maps using semantic segmentation networks. However, footwear products typically contain complex pattern structures and irregular edge features, such as interwoven eyelets, midsoles with gradient textures, or porous mesh uppers. These areas often lack clear binarization boundaries at the pixel level. Conventional masking techniques, if the edge delineation is not precise enough, or if the segmentation network deviates from recognizing sketches from non-standard perspectives, will result in jagged edges, artifacts, or structural breaks in the fused image at the joints of components, failing to meet the requirements of industrial design for realism and structural continuity.

[0086] This scheme first encodes and combines the textual semantic features of the target component with the visual features of the source reference image and hybrid weights during the data preprocessing stage to construct a tightly coupled element feature vector. Then, during the inference process of the generative model, a cross-attention mechanism is used to parse the aforementioned element feature vector. Since the cross-attention mechanism calculates the correlation between the input conditions and the intermediate features of the generated image, the textual semantic components in the element feature vector can activate the corresponding spatial attention map within the generative model. This allows for the automatic determination of the probability distribution region of the target component on the generated image plane based on semantic understanding; for example, the semantic information of "midsole" automatically leads to a high response in the lower part of the image. This spatial attention map, activated by textual semantics, constitutes an implicit, soft spatial distribution constraint. This application utilizes this constraint to guide the visual features in the element feature vector to be injected only into the high-response spatial region, thereby achieving automatic alignment of visual features and semantic regions without the need for explicit input coordinates or masks. Unlike conventional hard mask clipping, the response mapping generated by the cross-attention mechanism is a continuous probability distribution. This allows component features from different sources to smoothly transition at the boundaries according to the statistical laws of image generation, preserving the natural gaps and light and shadow transitions of the footwear product. Meanwhile, by directly introducing hybrid weights when constructing the element conditional sequence, the fusion of multi-source features occurs in the hidden feature space rather than the pixel space. The model can complete the mathematical aggregation of multi-dimensional features before feature decoding based on the weights, thus presenting a fusion effect with a rigorous structure and natural texture in the final output design.

[0087] As one aspect of this embodiment, by establishing a mapping relationship between entity words in natural language instructions and a preset set of standard components, semantic ambiguity in non-standardized user descriptions is eliminated, ensuring that the generated model can accurately correspond to the specific structure of footwear. Simultaneously, when the same component involves multiple sources, the initial weight values ​​are parsed separately, enabling the model to quantify and distinguish the feature contributions of different sources in subsequent processing. This achieves precise proportional control during the feature mixing stage, avoiding confusion between primary and secondary features due to unclear weight allocation.

[0088] As one aspect of this embodiment, high-dimensional hidden layer features and color palette style vectors of the source reference image are extracted respectively, and they are concatenated with text embedding vectors and mixing weights, so that the generation model can independently acquire and utilize texture details and color distribution, preventing color deviation or texture blurring caused by feature entanglement during the fusion process, and ensuring that the generated part can accurately reproduce the expected color style while maintaining the material texture of the reference image.

[0089] As one aspect of this embodiment, a multilayer perceptron is used to project the spliced ​​element feature vectors into dimensions, mapping the composite vector containing multidimensional information to the key-value pair dimension that the model's cross-attention layer can accept. This ensures that externally injected control signals can participate in the model's attention calculation, guaranteeing the smooth flow of feature injection channels and numerical stability during the model's inference process.

[0090] As one aspect of this embodiment, a normalized exponential function temperature adjustment process is introduced when processing multi-source feature fusion. This allows the original weights to be smoothed or sharpened according to the temperature parameter, thereby adjusting the blending smoothness of different source features during the feature aggregation stage. This avoids the feature of a certain source being over-amplified or suppressed due to the linear difference in the original weight values, so that the final generated component can more naturally present the expected blending effect of multiple reference map features.

[0091] As one aspect of this embodiment, the spatial attention map activated by text semantic features is used as an explicit spatial distribution constraint, which forces visual features to take effect only within the pixel area corresponding to the semantic logic. This effectively suppresses the accidental overflow of texture features into the background area or adjacent component area, thereby significantly improving the boundary clarity and positional accuracy of local feature migration without human intervention.

[0092] As one aspect of this embodiment, a multi-dimensional scoring module is integrated, which includes four dimensions: structural rationality, style consistency, color coordination, and material authenticity. This allows for the automatic identification and elimination of inferior images with structural distortion, stylistic inconsistencies, or material distortions from the randomly distributed results output by the generated model. This ensures that the final output design drawings meet industrial design standards in both aesthetics and physical logic, reducing the time cost for users to select effective solutions from a large number of candidate images.

[0093] As one aspect of this embodiment, a conflict detection logic based on cosine similarity and structural proportion parameters is executed in the scoring module to automatically detect illusory images that, although clear, deviate significantly from the features of the reference image or violate the basic structural proportions of footwear, thus ensuring that the fused design drawings are feasible for industrial applications.

[0094] As one aspect of this embodiment, targeted text optimization suggestions are generated based on the scoring results. By analyzing the reasons for low scores, specific adjustment directions are provided to users, helping them quickly locate the root cause of poor generation results. This can shorten the design cycle of shoe fusion and thus improve design efficiency.

[0095] The foregoing description of the specifications and embodiments is intended to explain the scope of protection of this invention, but does not constitute a limitation on the scope of protection of this invention. Modifications, equivalent substitutions, or other improvements to the embodiments of this invention or a portion thereof that can be obtained by those skilled in the art through logical analysis, reasoning, or limited experimentation, based on the teachings of this invention or the foregoing embodiments, in conjunction with common knowledge, general technical knowledge, and / or existing technology, should all be included within the scope of protection of this invention.

Claims

1. A method for generating shoe designs based on the fusion of text and images, characterized in that, Includes the following steps: Acquire user design input data, which includes at least one reference image and natural language instructions; Semantic parsing is performed on the natural language instructions to determine the target component to be generated, the source reference image index corresponding to each target component, and the mixing weight; An element condition sequence is constructed based on the design input data, wherein, for each target component, the visual features of the corresponding source reference image are extracted, and the visual features are encoded and combined with the text semantic features of the target component and the hybrid weight to obtain the element feature vector in the element condition sequence. The element conditional sequence is input into the generation model, and a semantic response mapping between the element feature vector and the spatial region of the generated image is established using a cross-attention mechanism. The visual representation of each target component in the generated image is controlled according to the semantic response mapping. Obtain at least one candidate fusion image output by the generative model, and output the final design image.

2. The method for generating shoe designs based on text and images according to claim 1, characterized in that, The semantic parsing of the natural language instructions includes: The entity words in the natural language instruction are identified using a natural language processing model, and the entity words are mapped to categories in a preset set of standard components, which includes footwear component categories. When the natural language instruction specifies multiple source reference images for the same target component, the initial weight value corresponding to each source reference image is parsed respectively.

3. The method for generating shoe designs based on text and images according to claim 1, characterized in that, The step of extracting the visual features of the corresponding source reference image includes: High-dimensional hidden layer features of the source reference image are extracted using an image coding network; The color palette style vector of the source reference image is extracted using a color extraction algorithm or a style encoder; The element feature vector is composed of the text embedding vector of the target component, the high-dimensional hidden layer features of the source reference image, the mixing weights, and the color palette style vector.

4. The method for generating shoe designs based on text and images according to claim 3, characterized in that, The step of constructing the element condition sequence based on the design input data further includes: The concatenated element feature vector is dimensionally projected using a multilayer perceptron, so that the dimension of the element feature vector is consistent with the dimension of the key-value pairs in the cross-attention layer inside the generation model.

5. The method for generating shoe designs based on text and images according to claim 1, characterized in that, When the element conditional sequence is input into the pre-trained generative model, the processing procedure of the cross-attention mechanism includes: When the same target component corresponds to multiple source reference images, the mixed weights corresponding to the multiple source reference images are subjected to normalized exponential function temperature adjustment processing to obtain normalized weights; Based on the normalized weights, the visual features from different source reference images are weighted and aggregated to generate aggregated component control features.

6. The method for generating shoe designs based on text and images according to claim 1, characterized in that, The step of establishing a semantic response mapping between the element feature vector and the spatial region of the generated image using a cross-attention mechanism includes: The spatial attention map inside the generation model is activated using the textual semantic features in the element feature vector to determine the distribution area of ​​the target component in the generated image; Using the spatial attention map as a spatial distribution constraint, the visual features in the element feature vector are guided to be injected into the distribution area, thus completing the local feature transfer.

7. The method for generating shoe designs based on text and images according to claim 1, characterized in that, The steps for outputting the final design drawing include: The generative model is controlled to generate multiple candidate fused images; The candidate fused images are input into a preset multi-dimensional scoring module to calculate the scores of each candidate fused image in terms of structural rationality, style consistency, color coordination, and material realism. The candidate fused images are sorted according to the scores, and the image with the highest score is selected as the final design image for output.

8. The method for generating shoe designs based on text and images according to claim 7, characterized in that, The multidimensional scoring module executes conflict detection logic, including: Calculate the cosine similarity between the feature vector of the candidate fused image and the feature vector of the weighted reference image; If the cosine similarity is lower than a preset threshold, or if the structural proportion parameter of the candidate fused image deviates from a preset proportion range, the score of the candidate fused image shall be reduced.

9. The method for generating shoe designs based on text and images according to claim 7, characterized in that, The final output design drawing also includes: Based on the scores from the multi-dimensional scoring module, text optimization suggestions for the final design drawing are generated and output; the text optimization suggestions include recommendations for adjusting the hybrid weights or material parameters.

10. A shoe design fusion generation system based on text and images, characterized in that, include: The input acquisition module is used to receive multiple reference images and natural language commands; The parsing and processing module is used to perform semantic parsing on the natural language instructions, determine the target component, the source reference image index and the mixing weight, and extract the visual features of the reference image; The sequence construction module is used to construct an element condition sequence based on the parsing results. The element condition sequence includes an element feature vector that integrates text semantic features, visual features, and mixed weights. The fusion generation module is used to input the element condition sequence into the generation model and establish a semantic response mapping between the visual features and the spatial regions of the generated image through a cross-attention mechanism to generate candidate fused images. The preferred output module is used to evaluate and sort the candidate fused images and output the final design drawing.

Images