Sketch and text based garment image generation method and system

Abstract

The application discloses a kind of based on sketch and text's clothing image generation method and system.Utilize a kind of sketch priori embedding module to process input sketch, extract key information and ensure contour consistency, to enhance the detail performance of image.Then, in combination with a cross-space attention mechanism, by processing text information to solve the problem of text information loss, ensure the style consistency and overall coherence of image.In addition, in order to reduce the complexity of network and maintain image quality, a network simplification scheme is proposed.Experimental results show that the proposed method performs excellently in generating high-fidelity clothing images, better than existing multi-modal generation methods.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for generating clothing images based on sketches and text. Background Technology

[0002] In recent years, with the development of artificial intelligence (AI) technology, especially breakthroughs in deep learning, computer vision and multimodal generative techniques have been widely applied. Generative models such as Generative Adversarial Networks (GANs) and Diffusion Models (DMs) have shown great potential in tasks such as text-to-image generation and multimodal image generation, achieving significant progress, particularly in applications such as fashion design and virtual try-on. Text-to-image generation technology, as a cross-modal conversion task, has become a hot research direction in the fields of computer vision and multimedia.

[0003] In traditional fashion design, designers typically rely on hand-drawn sketches, fabric samples, and manual adjustments to complete their designs. However, with the development of deep generative models, AI technology is gradually changing this traditional process. Designers can automatically generate garment images based on text descriptions or sketches, significantly improving design efficiency and accuracy. Against this backdrop, text-to-image generation methods have garnered particular attention, aiming to automatically generate images that match the content of text descriptions.

[0004] Existing text-based image generation technologies, especially in fashion design, face several challenges: First, how to maintain the outline accuracy and detail of the sketch during image generation to avoid shape distortion or inconsistent style in the generated image; second, how to effectively integrate textual information with image structure in the case of multimodal inputs (such as sketches and text), ensuring that the image content is semantically consistent with the input text and that the style of each part of the image is consistent. Furthermore, existing generative models often suffer from high complexity; how to reduce model complexity and computational cost while maintaining the quality of the generated image is also a technical challenge that needs to be addressed. Summary of the Invention

[0005] This invention addresses the problems of inaccurate image contours, poor style consistency, and high model complexity in existing technologies by proposing a method and system for generating clothing images based on sketches and text. This method introduces a sketch prior embedding module and a structure-aware multi-scale attention mechanism to extract and fuse key structural and semantic features from sketches and text descriptions, thereby achieving accurate reconstruction of sketch contours and stylistic consistency of text information. Furthermore, by introducing a variance-guided network simplification strategy into the generation network, this invention effectively improves the quality, stability, and computational efficiency of the generated images, ensuring the detail fidelity and stylistic consistency of the generated clothing images.

[0006] The technical solution of this invention is a method for generating clothing images from sketches and text, and controlling the consistency of their outlines and styles, comprising the following steps:

[0007] Step 1: Input textual information describing the clothing and encode the input text into a feature matrix;

[0008] Step 2: Input a sketch image of the clothing, and extract features from the input sketch using the sketch prior embedding module;

[0009] Step 3: The input text feature matrix and the features of the sketch image are fused and input into the UNet encoder to generate preliminary clothing image features;

[0010] Step 4: Utilize the structure-aware multi-scale attention mechanism to further process sketch and text features by calculating the correlation between different parts of the feature map, thereby improving the outline and style consistency of the clothing image.

[0011] Step 5: The results obtained in Step 4 are further processed using zero convolution to extract more accurate features. The zero convolution is reduced by using variance-guided network simplification.

[0012] Step 6: Randomly sample Gaussian noise from a Gaussian distribution to generate a new noisy image, and input this noise image into another UNet architecture identical to the encoder in Step 3. Simultaneously, the text features extracted in Step 1 are also conditionally input into the UNet to guide the generation process. During denoising, noise is gradually reduced and an intermediate image is generated by fusing the noise with the results extracted in Step 5. Through multiple iterations of denoising, a clear clothing image is finally obtained, which conforms to both the outline of the input sketch and the style of the text description.

[0013] Step 7: During the denoising process, the difference between the denoised image and the target image at each stage is calculated using Mean Squared Error Loss (MSE Loss). Specifically, MSE Loss measures the pixel-level error between the denoised image and the target image. The error gradient is propagated back to the model through backpropagation, thus obtaining progressively optimized gradient information. This gradient information is used to update the weight parameters of the UNet model, enabling the model to better reduce noise and generate intermediate results closer to the target image in each denoising step. Through multiple iterations of optimization, the UNet model gradually learns how to recover a clear clothing image that matches the sketch and text description from a noisy image.

[0014] Furthermore, the specific process of step 2 is as follows: based on the input sketch S inputFirst, a preliminary feature extraction module captures basic edge features. This module includes a first convolutional layer that extracts edge features and processes them through batch normalization and the ReLU activation function to obtain a preliminary feature map S1. Subsequently, a second convolutional layer further refines the features to obtain a refined feature map S2. Feature map S2 captures the key features in the sketch, which is crucial for subsequent processing. This part of the structure enables the model to better capture and process important features in the input sketch. To remove redundant information, a global feature weighting module then performs global pooling on S2 to obtain a global feature vector S. gap :

[0015]

[0016] Among them, S gap S represents the output of the global average pooling layer. 2c Let c represent the c-th channel of S2, where c ∈ [1, C].

[0017] The channel weights are calculated using a fully connected layer and a sigmoid activation function to obtain the weighted feature map S. g These weighting factors are used to adjust the importance of each channel, thereby highlighting important features and suppressing unnecessary parts:

[0018] S g =σ(U2δ(U1S) gap ))

[0019] Where δ represents the Silu activation function, σ represents the Sigmoid function, U1 is the weight matrix used to reduce the channel dimension, U2 is the weight matrix used to restore the channel dimension, and r is the scaling parameter, which aims to reduce the number of channels, thereby reducing the amount of computation.

[0020] Finally, to prevent feature information from gradually disappearing or being over-transformed in deep networks, a residual connection mechanism is used to fuse the weighted feature map with the initial feature map, resulting in a stable feature map S. res Through downsampling and convolution processing, the final output S is obtained. out This provides stable structural features for subsequent image generation, and the process is as follows:

[0021] S res =S2⊙S g ,

[0022]

[0023] Where ⊙ represents the element-wise Hadamard product. This indicates a downsampling operation, and β indicates normalization. This indicates a convolution operation.

[0024] Furthermore, the specific process of step 4 is as follows:

[0025] Based on the features output in step 3, and further processed through self-attention and cross-attention mechanisms to obtain feature map h, feature map h is input into the structure-aware multi-scale attention mechanism. First, the feature map is split along the channel dimension into multiple sub-feature maps h = [h0, h1, ..., h...]. G-1 Each sub-feature map focuses on a different spatial region. Then, a global pooling operation is performed to extract global context information, resulting in the global feature dependency h. x and h y Furthermore, a globally weighted feature map h is obtained by adjusting the importance of each channel through 1×1 convolution and a sigmoid activation function. spatial :

[0026]

[0027] Where d represents the global feature of the d-th channel of the feature map, d∈[1,K / G], h d (i,j) represents the value at position (i,j), h x and h y Let represent the average pooling results in the horizontal and vertical directions, respectively. σ is the sigmoid activation function, and W represents the weights of the 1×1 convolutional kernel. This indicates a splicing operation on the channel dimension.

[0028] Simultaneously, 3×3 convolution is used to extract local features h. local Next, spatial attention weights α1 and α2 are calculated using a cross-spatial mechanism. This mechanism allows the model to focus on key spatial regions, thereby improving its ability to capture long-range dependencies and enhancing feature consistency. Using these spatial attention weights to optimize spatial features makes the generated images more coherent. This step strengthens the model's understanding of the relationships between different regions in the feature map, thus contributing to improved overall consistency.

[0029]

[0030] α1 = Softmax(AvgPool(h) local ),α2=Softmax(AvgPool(h spatial ))

[0031] Among them, h local W represents the locally extracted features. 3×3 This represents a 3×3 convolution operation.

[0032] Combine the weight α1 with the global weighted feature map h spatialh is obtained by weighting the feature map through element-wise multiplication. out1 Similarly, weight α2 is combined with local feature map h local Dot product yields h out2 Then feature h out1 and h out2 The feature map h is obtained by fusion and addition. out To enhance the overall consistency and detail preservation of the generated images, the following measures are taken:

[0033] h out1 =α1·h spatial ,h out2 =α2·h local ,h out =σ(h) out2 +h out1 )

[0034] Among them, h out This is the final output feature map, where σ is the Sigmoid activation function, and h... out1 and h out2 These represent the feature map outputs based on spatial attention weights α1 and α2, respectively.

[0035] Furthermore, the specific process of step 5:

[0036] For each coding block output in step 4, the feature h out These features are further processed through zero-convolution to extract additional features. To measure the dispersion of the output features, we compute the direct output feature h of each coded block. out The variance of the features after zero convolution is calculated and referred to as the left variance. Furthermore, we calculate the variance of the feature output after zero convolution and define it as the right variance. We use the following formula to determine the effect of zero convolution on each encoded block:

[0037]

[0038] Where, σ L σ represents the left variance. R Let denot be the right variance and η be the variance reduction. This allows us to measure the variance change caused by zero convolution for each layer. For each variance reduction η obtained, we sort them in ascending order and retain the six zero convolution layers with the smallest variance reduction, starting from the layer with the smallest variance reduction.

[0039] The present invention also provides a clothing image generation system based on sketches and text, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the clothing image generation method based on sketches and text as described in the above technical solution.

[0040] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0041] 1. Accurately preserve sketch outline information: The sketch prior embedding module designed in this invention can better capture the key outlines and structural features in the sketch, and the generated clothing image can accurately reflect the outlines and details in the sketch.

[0042] 2. Style Consistency: By introducing a structure-aware multi-scale attention mechanism, this invention can maintain style consistency in different parts of the clothing during the generation process, effectively avoiding the problem of inconsistent style in traditional methods.

[0043] 3. Reduced model complexity: By employing a variance-guided network simplification strategy, this invention effectively reduces network complexity and improves model efficiency while ensuring the quality of generated images and the stability of results. Attached Figure Description

[0044] Figure 1 This is an overall flowchart of the present invention for generating clothing images from sketches and text.

[0045] Figure 2 This is a flowchart illustrating the prior embedding module of the present invention.

[0046] Figure 3 This is a schematic diagram of the structure-aware multi-scale attention mechanism of the present invention.

[0047] Figure 4 This is a schematic diagram of the variance-guided network simplification strategy process of the present invention.

[0048] Figure 5 This is a schematic diagram comparing the generation results of the method of the present invention with those of other multimodal methods.

[0049] Figure 6 This is a schematic diagram comparing the generation results of the present invention with and without the sketch prior embedding module.

[0050] Figure 7 This is a schematic diagram comparing the generation results of the present invention with and without the structure-aware multi-scale attention mechanism.

[0051] Figure 8 This is a comparative schematic diagram of the present invention with and without the variance-guided network simplification strategy. Detailed Implementation

[0052] The present invention will be further described below with reference to specific embodiments and accompanying drawings. These embodiments are provided to enable those skilled in the art to better understand the invention and do not constitute any limitation thereof.

[0053] like Figure 1As shown, the clothing image generation method based on sketches and text provided in this embodiment of the invention includes the following steps:

[0054] Step 1: Input text information describing the clothing, and use the clip text encoder to encode the input text into a feature matrix;

[0055] Step 2: Input a sketch image of the clothing, and extract features from the input sketch using the sketch prior embedding module;

[0056] Step 3: The input text feature matrix and the features of the sketch image are fused and input into the UNet encoder (SD Encoder) to generate preliminary clothing image features;

[0057] Step four: Utilize the structure-aware multi-scale attention mechanism to further process sketch and text features by calculating the correlation between different parts of the feature map, thereby improving the outline and style consistency of the clothing image.

[0058] Step 5: Using the results obtained in Step 4, further process them through zero convolution to extract more accurate features. The zero convolution is reduced by using variance-guided network simplification.

[0059] Step six involves randomly sampling Gaussian noise from a Gaussian distribution and using it as the initial noisy image input into another UNet architecture (including SD Encoder and SD Decoder) identical to the encoder in step three. During denoising, noise is progressively reduced and an intermediate image is generated by fusing the noise with the result extracted in step five. Through multiple iterations of denoising, a clear clothing image is finally obtained, conforming to both the outline of the input sketch and the style of the text description.

[0060] Step seven involves calculating the difference between the denoised image and the target image at each stage using Mean Squared Error Loss (MSE Loss). Specifically, MSE Loss measures the pixel-level error between the denoised image and the target image. The error gradient is then propagated back to the model via backpropagation to obtain progressively optimized gradient information. This gradient information is used to update the weight parameters of the UNet model, enabling it to better reduce noise and generate intermediate results closer to the target image at each denoising step. Through multiple iterations of optimization, the UNet model gradually learns how to recover a clear clothing image that matches the sketch and text description from a noisy image.

[0061] Based on the above methods, the network designed in this invention is as follows: Figure 1 As shown, this includes a sketch prior embedding module and a structure-aware multi-scale attention mechanism. The sketch prior embedding module process includes the following steps:

[0062] Step 11: Input the sketch image S and process it through the initial feature extraction module. First, the first convolutional layer extracts edge features, followed by batch normalization and ReLU activation to obtain feature map S1. Then, the second convolutional layer further refines the features, and after BN2, a refined feature map S2 is obtained, which is used to capture key information in the sketch.

[0063] Step 12: Based on the feature map S2 obtained in the previous step, we designed a global feature weighting submodule. This submodule first performs global pooling on S2, compressing the spatial dimension into a global feature vector S. gap The specific implementation is as follows:

[0064]

[0065] Where S 2c Let S represent the c-th channel of S2, where c∈[1,C], and H, W, and C represent the height, width, and number of channels of the feature map, respectively.

[0066] Step 13: In order to model complex feature relationships and capture key information in the sketch, the global feature vector S is... gap The feature map S is processed through a fully connected layer and a Sigmoid activation function. Then, a weight factor for each channel is calculated using a fully connected layer and a Sigmoid activation function to obtain the feature map S. g These weighting factors are used to adjust the importance of each channel, highlighting key features and suppressing redundant parts. The specific process is as follows:

[0067] S g =σ(U2δ(U1S) gap ))

[0068] Where δ represents the Silu activation function, σ represents the Sigmoid function, U1 is the weight matrix used to reduce the channel dimension, U2 is the weight matrix used to restore the channel dimension, and r is a scaling parameter used to reduce the number of channels and reduce the computational cost.

[0069] Step 14: To prevent feature information from gradually disappearing in deep networks, we introduce the ResNet residual connection mechanism to fuse the weighted feature map with the initial feature map, obtaining a stable feature map S. res Through downsampling and convolution processing, the final output feature S is obtained. out The specific process is as follows:

[0070] S res =S2⊙S g ,

[0071]

[0072] Where ⊙ represents the element-wise Hadamard product. This indicates a downsampling operation, and β indicates normalization. This indicates a convolution operation.

[0073] The process of structure-aware multi-scale attention mechanisms includes the following steps:

[0074] Step 15, using the feature S output in step 14 out The input text feature matrix is ​​compared with the feature S of the sketch image. out The fused input is fed into the UNet encoder to generate preliminary clothing image features, resulting in feature h. At this point, the feature map h is segmented into G sub-features h = [h0, h1, ..., h...]. G-1 ], each sub-feature h i The size is M×N×K / G, where M, N, and K represent the sub-feature h, respectively. i The height, width, and number of channels are considered. Each sub-feature focuses on a different spatial region, helping the model understand the multidimensional features of the input, thereby improving the overall consistency and detail of the generated clothing images.

[0075] Step 16: We construct a global pooling submodule to extract global context information from the feature map. For each sub-feature h in step 15... i Global context is extracted by performing average pooling operations along the spatial dimensions (horizontal and vertical). This operation compresses spatial information into a single global feature for each channel, helping the model capture long-range dependencies. The specific implementation is as follows:

[0076]

[0077] Where d represents the global feature of the d-th channel of the feature map, d∈[1,K / G], h d (i,j) represents the value at position (i,j), h x and h y These represent the average pooling results in the horizontal and vertical directions, respectively, where σ is the Sigmoid activation function.

[0078] Step 17: To improve the model's ability to focus on more relevant features, extract global features h... x and h y Then, they are concatenated and processed through a 1×1 convolution, followed by the computation of weights for each channel using a sigmoid activation function. These weights are used to adjust the importance of each channel in the feature map according to the global context. The specific process is as follows:

[0079]

[0080] Among them, hspatial Let W represent the weight matrix for each channel, where σ is the sigmoid activation function and W is the weight of the 1×1 convolution kernel. This indicates a splicing operation along the channel dimension.

[0081] Step 18: Extract local features from the original feature map h using 3×3 convolution to capture fine-grained spatial details and supplement the global context. The specific process is as follows:

[0082]

[0083] Among them, h local W represents the locally extracted features. 3×3 This represents a 3×3 convolution operation.

[0084] Step 19 involves calculating spatial attention weights α1 and α2 using a cross-spatial mechanism. This mechanism enables the model to focus on key spatial regions, thereby improving its ability to capture long-range dependencies and enhancing feature consistency. Using these spatial attention weights to optimize spatial features results in a more coherent image. This step strengthens the model's understanding of the relationships between different regions in the feature map, thus contributing to improved overall consistency. Specifically:

[0085] α1 = Softmax(AvgPool(h) local ),α2=Softmax(AvgPool(h spatial ))

[0086] Here, AvgPool represents average pooling, and Softmax represents the normalized activation function.

[0087] Step 110: Finally, by optimizing the weighted combination of feature maps, the detail representation and overall consistency of the generated image are further improved. This is achieved by combining weights α1 and α2 with the weighted feature map h. spatial and local feature map h local The optimized feature map h is obtained out To enhance the overall consistency and detail preservation of the generated images, the following measures are taken:

[0088] h out1 =α1·h spatial ,h out2 =α2·h local ,h out =σ(h) out2 +h out1 )

[0089] Among them, h out This is the final output feature map, where σ is the Sigmoid activation function.

[0090] The variance-guided network simplification strategy is as follows:

[0091] Step 111: For the feature h output by each coding block in step 110 out These features are further processed through zero-convolution to extract additional features. To measure the dispersion of the output features, we compute the direct output feature h of each coded block. out The variance of the features after zero convolution is calculated and referred to as the left variance. Furthermore, we calculate the variance of the feature output after zero convolution and define it as the right variance. We use the following formula to determine the effect of zero convolution on each encoded block, as follows:

[0092]

[0093] Where, σ L σ represents the left variance. R Let denot be the right variance and η be the amount of variance reduction. This allows us to measure the variance change caused by zero convolution for each layer.

[0094] Step 112: For each variance reduction η obtained, we first sort them in ascending order. Then, we compare the effects of the layer with the smallest variance reduction with the layer with the largest variance reduction. Experimental results show that the layer with the smallest variance reduction performs better while preserving zero convolutions. Based on this finding, we start with the layer with the smallest variance reduction and add other layers sequentially until the overall effect is close to the performance of the original model. Finally, we retain the six layers with the smallest variance reduction, which exhibit the smallest zero convolution effect while maintaining the model's performance.

[0095] On the other hand, embodiments of the present invention also provide a clothing image generation system based on sketches and text, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the clothing image generation method based on sketches and text as described in the above technical solution.

[0096] The effects of the present invention are illustrated below through specific embodiments:

[0097] 1. Implementation details

[0098] We used Stable Diffusion v1.5 as our base text-to-image generation model, which contains 16 cross-attention layers. During training, we used the AdamW optimizer based on PyTorch Lightning. The learning rate was set to 1×10^-5. Our model was trained on an NVIDIA RTX 4090-24G GPU with a batch size of 4. Both input and output images were 512×512 pixels.

[0099] 2. Data Preparation

[0100] We collected a new dataset. First, we gathered approximately 30,000 fashion images from multiple websites, along with corresponding text descriptions. To ensure consistency in image size and quality, we cropped and resized all collected images. Next, we used the Photosketch network to convert all fashion images into sketches. To further improve the clarity and usability of the sketches, we performed binarization to remove background noise and irrelevant details, thus simplifying the sketches. Each pair of data includes an original image, a corresponding sketch, and a detailed text description, all resized to 512×512 pixels.

[0101] 3. Evaluation Indicators

[0102] ●Frechet Inception Distance (FID): FID is a metric that measures the difference between a generated image and a real image. It is calculated by comparing the distribution differences between the generated and real images in the deep feature space. The smaller the FID value, the closer the feature distribution of the generated image is to that of the real image, and the better the generation effect. It is widely used to evaluate the performance of generative models such as generative adversarial networks.

[0103] ●SSIM (Structural Similarity Index): SSIM is an image quality assessment standard that evaluates the similarity between a generated image and a real image in terms of brightness, contrast, and structure. Its value ranges from 0 to 1, with higher values ​​indicating greater similarity and better quality. SSIM effectively reflects the visual perception of an image and is commonly used in image generation.

[0104] ●LPIPS (Learned Perceptual Image Patch Similarity): LPIPS is a perceptual image similarity evaluation method that extracts high-level features of images through deep neural networks to measure the perceptual difference between generated and real images. The lower the LPIPS value, the better the performance.

[0105] 4. Comparison with other models

[0106] Quantitative Evaluation: The experiment selected four state-of-the-art models for generating images under sketch and text conditions: ControlNet, Gligen, UniControl, and FineControlNet. To ensure a fair comparison with our method, we retrained all selected models on our custom dataset. By training these models on the same comparison dataset, we eliminated potential biases caused by data differences, ensuring that the evaluation results accurately reflect the model performance under the same conditions. We then quantified the comparison results, calculating the scores for LPIPS, FID, and SSIM. The metric results are shown in Table 1. As can be seen from the results in Table 1, our method significantly outperforms the comparison methods in terms of perceptual quality and feature alignment. The significant reductions in LPIPS and FID, as well as the improvement in SSIM, indicate that the fashion images generated by our method are not only perceptually closer to real images but also maintain more accurate feature distribution and structural integrity. This is crucial for the visual fidelity of the generated images and their consistency with the text description. The generated results are shown below. Figure 5 As shown, compared with other methods, the method of this invention not only reproduces the input sketch and text description excellently in terms of detail and accuracy, but also generates fashion images that are more visually consistent and diverse, demonstrating a significant advantage over competing methods. Thanks to the synergistic integration of the sketch prior embedding module and the structure-aware multi-scale attention mechanism, the method of this invention exhibits significant advantages in detail preservation and semantic consistency.

[0107] Table 1. Quantitative Comparison of Multimodal Text Conditional Image Generation Models

[0108] ControlNet 22.47 0.7007 0.2612 Gligen 16.29 0.7488 0.2119 UniControl 18.86 0.7429 0.2143 FineControlNet 19.39 0.7227 0.2255 This invention 15.31 0.7644 0.1966

[0109] 5. Ablation test

[0110] This section focuses on the impact of the proposed sketch prior embedding module, structure-aware multi-scale attention mechanism, and variance-guided network simplification on model efficiency and performance. To validate the contribution of each proposed module, we sequentially removed these modules from the entire model. The results are shown in Table 2, where w / o(SAMA) indicates the removal of the structure-aware multi-scale attention (SAMA) module, and w / o(SPEM) indicates the removal of the sketch prior embedding module (SPEM).

[0111] Table 2 Ablation Experiment

[0112] ControlNet 22.47 0.7007 0.2612 w / o SAMA 16.87 0.7601 0.2024 w / o SPEM 18.48 0.7378 0.2015 This invention (w / o VNS) 15.07 0.7685 0.1910

[0113] The impact of the sketch prior embedding module on image contours: As shown in the second row of Table 2, the sketch prior embedding module is crucial for preserving the fine details of the image contours. Removing this module results in the generated image failing to closely align with the input sketch, particularly showing distortion in key structural areas such as garment hems and cuffs. Figure 6 As shown, the lack of SPEM leads to shape inconsistencies and feature alignment issues, resulting in a significant decrease in the fidelity of the overall contour. This demonstrates that SPEM plays a crucial role in maintaining contour accuracy, ensuring that the generated image matches the input sketch.

[0114] The impact of structure-aware multi-scale attention on image style consistency: As shown in the third row of Table 2, structure-aware multi-scale attention enhances the model's ability to maintain style consistency across different image regions while effectively preserving textual information. From Figure 7 The examples provided illustrate that removing SAMA resulted in noticeable inconsistencies in key style elements such as texture and color distribution. For instance, in... Figure 7 In the first set of images, even though the text description is "pure black," removing SAMA results in a loss of textual information, and the generated clothing images do not match the text description. Furthermore, in the second set of examples, the style of the clothing images lacks consistency when SAMA is removed. Conversely, adding SAMA ensures a smooth style transition in the generated clothing images and aligns them with the text description of the generated images. In summary, the lack of a multi-scale attention mechanism reduces the model's ability to align textual information and coordinate global and local features, leading to a decrease in the coherence and realism of clothing style representation.

[0115] Impact of Variance-Guided Network Simplification on Model Efficiency and Performance: Table 3 shows a comparison of inference times, highlighting the significant impact of the Variance-Guided Network Simplification (VNS) module on improving computational efficiency. By introducing variance-guided network simplification, this module effectively reduces unnecessary zero-convolutions while preserving key features. This optimization reduces inference time by approximately 10%, from 6.73 seconds (without VNS) to 6.05 seconds (with VNS). It is worth noting that, as... Figure 8 As shown, the images generated with and without VNS show almost no visual difference, and their FID scores are also very close, reaching 15.31. This further demonstrates that VNS successfully balances computational efficiency while maintaining high-quality generated results. Specifically, the time savings prove the effectiveness of selecting the layer with the minimum variance reduction for simplification, thus achieving a balance between performance and efficiency. This efficiency improvement is achieved without affecting the quality of the generated output. These results validate the role of VNS in optimizing computational overhead while maintaining high model performance and provide a practical solution for scenarios requiring a balance between speed and accuracy.

[0116] Table 3 Comparison of inference time with and without the Variance Guided Network Simplification (VNS) module.

[0117] Time (s) 6.73 6.05

[0118] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for generating clothing images based on sketches and text, characterized in that, Includes the following steps: Step 1: Input textual information describing the clothing and encode the input text into a feature matrix; Step 2: Input a sketch image of the clothing and extract features from the input sketch; Step 3: The input text feature matrix and the features of the sketch image are fused and input into the Unet encoder to generate preliminary clothing image features; Step 4: Utilize a structure-aware multi-scale attention mechanism to further process sketch and text features by calculating the correlation between different parts of the feature map; The specific process of step 4 is as follows: Based on the features output in step 3, and further processed through self-attention and cross-attention mechanisms to obtain feature map h, feature map h is input into the structure-aware multi-scale attention mechanism. First, the feature map is split along the channel dimension into G sub-feature maps h=[h0,h1,…,h…]. G−1 ], each sub-feature h i The size is M×N×K / G, where M, N, and K represent the sub-feature h, respectively. i The height, width, and number of channels are determined; then, a global pooling operation is performed to extract global context information, resulting in the global feature dependency h. x and h y Furthermore, a globally weighted feature map h is obtained by adjusting the importance of each channel through 1×1 convolution and a sigmoid activation function. spatial ; Simultaneously, 3×3 convolution is used to extract local features h. local Next, spatial attention weights α1 and α2 are calculated through a cross-spatial mechanism; ； ； Among them, h local W represents the locally extracted features. 3×3 This represents a 3×3 convolution operation. Indicates average pooling; Combine the weight α1 with the global weighted feature map h spatial h is obtained by weighting the feature map using dot product. out1 Similarly, weight α2 is combined with local feature map h local Dot product yields h out2 Then feature h out1 and h out2 The feature map h is obtained by fusion and addition. out The details are as follows: ； Among them, h out This is the final output feature map, where σ is the Sigmoid activation function, and h... out1 and h out2 These represent the feature map outputs weighted by spatial attention weights α1 and α2, respectively. Step 5: The results obtained in Step 4 are further processed using zero convolution to extract more accurate features. The zero convolution is reduced by using variance-guided network simplification. Step 6: Randomly sample Gaussian noise from the Gaussian distribution and input it as the initial noise image into another UNet model, the same as the encoder in Step 3. At the same time, the text features extracted in Step 1 are also input into the UNet model as conditions to guide the generation process. During the denoising process, the noise is gradually reduced and an intermediate image is generated by fusing it with the results extracted in Step 5. Through multiple iterations of denoising, a clear clothing image is finally obtained. Step 7: During the denoising process, calculate the difference between the denoised image and the target image at each stage, and recover a clear clothing image that matches the sketch and text description from the noisy image through the iteratively optimized Unet model.

2. The method for generating clothing images based on sketches and text as described in claim 1, characterized in that: In step 2, the sketch prior embedding module extracts features from the input sketch. The specific process is as follows: based on the input sketch S input First, a preliminary feature extraction module captures basic edge features. This module includes: a first convolutional layer extracting edge features and processing them through batch normalization and ReLU activation to obtain a preliminary feature map S1; subsequently, a second convolutional layer further refines the features to obtain a refined feature map S2; to remove redundant information, a global feature weighting module then performs global pooling on S2 to obtain a global feature vector S. gap : The channel weights are calculated using a fully connected layer and a sigmoid activation function to obtain the weighted feature map S. g ; Finally, by combining the residual connection mechanism, the weighted feature map and the preliminary feature map are fused to obtain a stable feature map S. res Through downsampling and convolution processing, the final output feature S is obtained. out .

3. The method for generating clothing images based on sketches and text as described in claim 2, characterized in that: Global feature vector S gap The calculation method is as follows: ； Among them, S gap S represents the output of the global average pooling layer. 2c Let S represent the c-th channel of S2, where c∈[1,C], and H, W, and C represent the height, width, and number of channels of the feature map, respectively.

4. The method for generating clothing images based on sketches and text as described in claim 2, characterized in that: Weighted feature map S g The calculation method is as follows: ； Where δ represents the Silu activation function, σ represents the Sigmoid function, U1 is the weight matrix used to reduce the channel dimension, and U2 is the weight matrix used to restore the channel dimension.

5. The method for generating clothing images based on sketches and text as described in claim 2, characterized in that: Output feature S out The calculation method is as follows: ； ； Where ⊙ represents the element-wise Hadamard product, ⇓ represents the downsampling operation, and β represents normalization. represents the convolution operation, and S represents the input sketch.

6. The method for generating clothing images based on sketches and text as described in claim 1, characterized in that: Global weighted feature map h spatial The calculation method is as follows: ； ； Where d represents the global feature of the d-th channel of the feature map, d∈[1,K / G], h d (i,j) represents the value at position (i,j), h x and h y Let represent the average pooling results in the horizontal and vertical directions, respectively. σ is the sigmoid activation function, W represents the weights of the 1×1 convolution kernel, and ⊕ represents the concatenation operation along the channel dimension; h spatial This represents the weight matrix for each channel.

7. The method for generating clothing images based on sketches and text as described in claim 1, characterized in that: The specific process of step 5: For each coding block output in step 4, the feature h out Calculate the direct output feature h of each coded block. out The variance of the feature output after zero convolution is calculated and called the left variance. Additionally, the variance of the feature output after zero convolution is calculated and defined as the right variance. The effect of zero convolution on each coding block is determined using the following formula: ； Where, σ L σ represents the left variance. R η represents the right-hand variance, and η represents the decrease in variance. For each variance reduction η obtained, sort them in ascending order, and starting from the layer with the smallest variance reduction, retain several zero convolutional layers with the smallest variance reduction.

8. The method for generating clothing images based on sketches and text as described in claim 7, characterized in that: The six zero-convolutional layers with the smallest variance reduction are retained.

9. A clothing image generation system based on sketches and text, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the stored instructions in the memory to execute the clothing image generation method based on sketches and text as described in any one of claims 1-8.

Images