Diffusion model embroidery generation method and device based on decoupling of aesthetics and process

Abstract

The application relates to an embroidery generation method and device based on an aesthetic and process decoupling diffusion model, applied to the technical field of image generation, and comprising the following steps: mounting a pre-trained common feature adapter in a deep bottleneck layer of a backbone network of a diffusion model, and mounting a pre-trained specific residual adapter in a shallow high-resolution layer; extracting aesthetic features such as composition, color matching and material gloss of a embroidery image through the common feature adapter, and extracting detailed features such as needle method direction, fiber texture and process topology of the embroidery image through the specific residual adapter, so that the aesthetic features and the detailed features are effectively separated, the problem of mutual interference of different knowledge in a traditional generation model is avoided, and the authenticity and stability of a generation result are improved.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, specifically to a diffusion model embroidery generation method and apparatus based on the decoupling of aesthetics and craftsmanship. Background Technology

[0002] Embroidery, as an outstanding representative of traditional handicrafts and intangible cultural heritage, not only carries profound historical and aesthetic value but also possesses unique physical characteristics. Unlike ordinary flat painting or dyeing techniques, embroidery is a three-dimensional line art created by piercing and extending threads through fabric with a needle. Utilizing artificial intelligence to stylize digital embroidery has become a core research direction for digital cultural heritage protection, creative clothing design, and the digital transformation of handicrafts.

[0003] Text-to-image generation technology, with diffusion models at its core, has become a new paradigm for visual art creation and the digital preservation of cultural heritage. Especially in the field of intangible cultural heritage, the digital reconstruction of complex craft textures, color systems, and cultural semantics through deep learning models not only enables the living transmission of precious skills but also provides endless possibilities for the modern design and innovative application of intangible cultural heritage elements.

[0004] Early customization research primarily focused on feature capture for single subjects: Textual Inversion (an efficient and lightweight text-to-image model personalization technique) maps visual concepts to pseudo-word vectors in a frozen model vocabulary; DreamBooth (a subject-driven text-to-image generation model personalization fine-tuning technique, whose core is to fine-tune a pre-trained diffusion model with a small number of example images, allowing the model to learn to bind unique identifiers to specific visual subjects, thereby generating high-quality images of that subject in different scenes, poses, and styles) achieves high-fidelity reconstruction of specific instances through a sparse fine-tuning strategy. However, when attempting to apply multiple fine-tuned models to the same scene through simple linear combination, the models often exhibit severe "concept forgetting" or "attribute leakage." To address this issue, a new approach has emerged that updates only the key projection matrix in the attention mechanism and constrains the mutual exclusivity of multiple concepts in the latent space through a joint training objective function, thus achieving closed-loop generation of multiple concepts to some extent.

[0005] With the popularization of LoRA (Low-Rank Adaptation) technology, how to efficiently integrate multiple LoRA modules trained for different concepts has become a new research hotspot. Mix-of-Show (decentralized low-rank adaptation framework) proposes a decentralized fine-tuning paradigm. By training LoRA on each independent concept and using gradient fusion and embedding rewriting techniques, it effectively alleviates the problems of identity loss and attribute interference when merging multiple concepts. Although Mix-of-Show performs excellently when dealing with different individual objects, it still follows the logic of "parallel decoupling" in essence. The concepts are at an equal semantic level, making it difficult to finely separate common knowledge from individual characteristics.

[0006] The diverse textures and stitches of embroidery art, along with macro-level aesthetic norms (such as composition and material a priori characteristics), constitute the commonalities of the field, while micro-level stitch details represent task-specific characteristics. This hierarchical distribution of features is widespread in visual tasks with fine-grained differences. Although existing generative methods excel at representing broad-ranging general scenes and art styles, when faced with complex crafts like embroidery where commonalities and specific characteristics are highly coupled, the lack of explicit hierarchical control mechanisms in its latent space often leads to a severe entanglement between the characteristics of "macro-aesthetic style" and "micro-craft details." Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a diffusion model embroidery generation method and apparatus based on the decoupling of aesthetics and craftsmanship, which aims to solve the problem in the prior art that when models face complex processes such as embroidery with highly coupled commonalities and characteristics, the lack of an explicit hierarchical control mechanism in their latent space often leads to a serious entanglement between the features of "macro aesthetic style" and "micro craftsmanship details".

[0008] According to a first aspect of the present invention, a diffusion model embroidery generation method based on aesthetic and technological decoupling is provided, the method comprising:

[0009] Obtain text prompts, which contain descriptive information about the target embroidery image; The text prompt words are input into a text encoder for encoding to obtain a text conditional feature vector; The text conditional feature vector is input into the diffusion model, and the target embroidery image is generated through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image.

[0010] Preferably, The training of the common representation adapter and the specific residual adapter includes: Obtain both a general embroidery style dataset and a specific process dataset; Obtain the true common characteristics of the general embroidery style dataset, and obtain the true microscopic process-specific characteristics of the specific process dataset; Obtain the first text prompt word of the general embroidery style dataset, and obtain the second text prompt word of the specific process dataset; Activate the common representation adapter of the deep bottleneck layer of the backbone network and freeze the specific residual adapter of the shallow high-resolution layer of the backbone network. The first text prompt word is input into the backbone network, and the predicted common representation features are output through the activated common representation adapter. The error between the predicted common representation features and the real common representation features is measured by a preset first loss function. The common representation adapter is iteratively trained until a preset iteration stopping condition is met, and a trained common representation adapter is obtained. Activate the specific residual adapter of the shallow high-resolution layer of the backbone network and freeze the trained common representation adapter; The second text prompt is input into the backbone network, and the predicted micro-process specific features are output through the activated specific residual adapter. The error between the predicted micro-process specific features and the actual micro-process specific features is measured by a preset second loss function. The specific residual adapter is iteratively trained until the preset iteration stopping condition is met, and the trained specific residual adapter is obtained.

[0011] Preferably, The acquisition of the general embroidery style dataset and the specific process dataset includes: A multimodal embroidery dataset is obtained, and the multimodal data is decoupled from the multimodal embroidery dataset. Images containing diverse cross-style embroidery images are assigned to the general embroidery style dataset, and images of embroidery images with specific needlework techniques are assigned to the specific technique dataset.

[0012] Preferably, The first loss function is the diffusion loss function; The second loss function is a joint loss function obtained by weighted summation of the structure tensor loss function and the spectrum preservation loss function.

[0013] Preferably, After obtaining the target embroidery image, the process further includes: The target embroidery image output by the decoder is input into the super-resolution reconstruction module, which performs a resolution magnification operation on the target embroidery image to obtain an ultra-high-definition embroidery image.

[0014] Preferably, After obtaining the ultra-high-definition embroidery image, the process also includes: The ultra-high-definition embroidery image is converted from the RGB color gamut to the industrial-grade CMYK color gamut or the Pantone special color gamut for intangible cultural heritage, and then subjected to noise reduction and contrast optimization processing to obtain an industrial-grade lossless embroidery image.

[0015] Preferably, The backbone network adopts the U-Net network, the deep bottleneck layer is Layers 3, 4, and 5 of the U-Net network, and the shallow high-resolution layer is Layers 1, 2, 6, and 7 of the U-Net network.

[0016] According to a second aspect of the present invention, a diffusion model embroidery generation apparatus based on aesthetic and technological decoupling is provided, the apparatus comprising: Text prompt word acquisition module: used to acquire text prompt words, which contain descriptive information about the target embroidery image; Text encoding module: used to input the text prompt words into the text encoder for encoding, and obtain the text conditional feature vector; Image generation module: used to input the text conditional feature vector into the diffusion model, and generate the target embroidery image through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image.

[0017] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This application employs a pre-trained common representation adapter mounted in the deep bottleneck layer of the backbone network of the diffusion model, and a pre-trained specific residual adapter mounted in the shallow high-resolution layer. The common representation adapter extracts aesthetic features such as composition, color scheme, and material gloss from the embroidery image, while the specific residual adapter extracts detailed features such as needlework direction, fiber texture, and process topology from the embroidery image. This effectively separates aesthetic features from detailed features, avoiding the problem of mutual interference between different knowledge in traditional generative models, thereby improving the realism and stability of the generated results.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 This is a flowchart illustrating an embroidery generation method based on a diffusion model decoupling aesthetics and craftsmanship, according to an exemplary embodiment. Figure 2 This is a schematic diagram of the overall framework according to another exemplary embodiment; Figure 3 This is a schematic diagram of a diffusion model embroidery generation device based on the decoupling of aesthetics and craftsmanship, according to another exemplary embodiment. In the attached diagram: 1-Text prompt word acquisition module, 2-Text encoding module, 3-Image generation module. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0022] Example 1: Figure 1 This is a flowchart illustrating an embroidery generation method based on a diffusion model decoupling aesthetics and craftsmanship, according to an exemplary embodiment. Figure 1 As shown, the method includes: S1, Obtain text prompts, the text prompts containing descriptive information about the target embroidery image; S2, input the text prompt words into the text encoder for encoding to obtain the text conditional feature vector; S3, input the text conditional feature vector into the diffusion model, and generate the target embroidery image through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image; Understandably, as shown in the attached document Figure 2 As shown, the natural language description of the intangible cultural heritage craft input by the user is obtained as the initial text prompt. For example, the user inputs descriptive information such as "Qinghai embroidery peony pattern, red silk thread, gold outline, traditional Tibetan style".

[0023] The text prompts are input into a text encoder to extract high-dimensional conditional embedding vectors, transforming the text description into a sequence of text feature vectors.

[0024] The extracted text conditional feature vectors are injected into the backbone network of the pre-trained diffusion model as semantic priors for subsequent diffusion denoising and fine-tuning guidance.

[0025] In this embodiment, the diffusion model can adopt the Stable Diffusion XL (SDXL) model. In addition to the SDXL model, other diffusion models such as Stable Diffusion 1.5, FLUX, PixArt-α, and DiT (Diffusion Transformer) can also be used as the basic generation framework. By injecting aesthetic prior features and process residual features at different levels, the hierarchical knowledge decoupling generation of embroidery images can be achieved. Although different models differ in terms of generation quality, inference speed and parameter scale, they can all achieve independent modeling and collaborative generation of embroidery aesthetics and process features.

[0026] The backbone network is a U-Net network, with common representation LoRA modules (CR-LoRA) mounted in its deep bottleneck layers (Layers 3, 4, and 5), and specific residual LoRA modules (SR-LoRA) mounted in its shallow high-resolution layers (Layers 1, 2, 6, and 7). It is worth emphasizing that, in addition to LoRA adapters, Adapter, Prefix-Tuning, Prompt-Tuning, IA, or other parameter-efficient fine-tuning techniques can be used to establish common representation adapters and specific residual adapters at different levels of the generative model. Through parameter isolation and staged training strategies, decoupling learning of embroidery aesthetic features and craft texture features is achieved, thereby achieving a generative effect similar to that of this application.

[0027] The Common Representation LoRA module is used to represent the global aesthetic common features of embroidery, including global material, color distribution and macroscopic aesthetic a priori features; the Specific Residual LoRA module is used to represent the microscopic process-specific features of embroidery, including microscopic stitch topology and the direction details of specific stitches (such as coiled embroidery and piled embroidery).

[0028] After inputting the text conditional feature vector into the U-Net network, the common representation feature (CR Layer Output) and the specific residual feature (SR Layer Residual) are output through the common representation LoRA module and the specific residual LoRA module, respectively. The common characterization features and specific residual features extracted by the aforementioned decoupling are simultaneously used as parallel input sources and input into the Nonlinear Structured Fusion module.

[0029] The fusion module uses a nonlinear manifold aggregation algorithm to perform high-order fusion of macroscopic aesthetic prior features and microscopic line trace residual features in manifold space.

[0030] The fused and optimized complete features are denoised through multiple rounds of iterative processes using a diffusion model, and the latent spatial representation is rendered and parsed into the initial embroidery raster image through its built-in variational autoencoder (VAE) decoder.

[0031] The decoded initial embroidery image is input into the super-resolution reconstruction module. While keeping the micro-stitches and high-frequency textures of the silk threads clear, a lossless ultra-high-definition physical magnification of 4× or 8× is performed to deeply reconstruct the micro-pixel details of the embroidery image.

[0032] The ultra-high-definition image is input into the color management and CMYK color gamut conversion module. Through a lossless color gamut mapping algorithm, the image is accurately converted from the RGB color gamut of the display to the industrial-grade CMYK color gamut or the Pantone special color library used by digital printing / digital weaving equipment, eliminating cross-device color bias.

[0033] Perform advanced noise reduction and contrast enhancement optimization, and automatically convert the final high-fidelity embroidery image into an industrial-grade lossless raster format (such as TIFF, RAW or high-quality PNG) for storage and output.

[0034] The generated results are subjected to feature encoding, parameter mapping, and format conversion operations to generate resource files containing image data, process description information, and model configuration parameters. By establishing a mapping relationship between the generated images and process features, the classification management and rapid retrieval of digital resources for different embroidery techniques are realized, providing technical support for subsequent embroidery design reuse, style transfer, and digital display.

[0035] It is worth emphasizing that the training of the backbone network described above in this embodiment adopts a hierarchical knowledge decoupling and two-stage training method, specifically including: To prepare multimodal data on Qinghai embroidery for model training, explicit decoupling was implemented, dividing the dataset into a general embroidery dataset containing diverse and cross-style embroidery techniques and a specific craft dataset focusing on vertical stitches. The true common characteristics of a general embroidery style dataset and the true micro-process-specific characteristics of a specific process dataset are obtained for subsequent loss value calculation. The feature extraction tools or methods are existing conventional technologies, which will not be elaborated on in this embodiment. Obtain the first text prompt words for a general embroidery style dataset of natural language descriptions of intangible cultural heritage crafts input by the user, and the second text prompt words for a specific craft dataset; Activate the common representation adapter of the deep bottleneck layer of the backbone network and freeze the specific residual adapter of the shallow high-resolution layer of the backbone network (freezing means not participating in parameter updates during the training process). The first text prompt word is input into the backbone network, and the predicted common representation features are output through the activated common representation adapter. The error between the predicted common representation features and the true common representation features is measured by a preset first loss function. The common representation adapter is iteratively trained until the change in the loss value in N consecutive iterations is lower than the preset change threshold or the preset number of iterations is reached, and the trained common representation adapter is obtained. In summary, during the common learning phase, a common prompt word is introduced in conjunction with a common embroidery style dataset. Deep fine-tuning is then performed in the backbone network. At this point, the parameter update strategy only activates the common representation LoRA module (CR-LoRA) mounted in the deep bottleneck layers (Layers 3, 4, 5), while completely freezing the high-resolution edge layers, forcing them to precipitate the global material, color distribution, and macroscopic aesthetic priors of the embroidery.

[0036] Activate the specific residual adapters of the shallow high-resolution layers of the backbone network and freeze the trained common representation adapters. The second text prompt is input into the backbone network, and the activated specific residual adapter outputs the predicted micro-process specific features. The error between the predicted micro-process specific features and the actual micro-process specific features is measured by a preset second loss function. The specific residual adapter is iteratively trained until the change in the loss value in N consecutive iterations is lower than the preset change threshold or the preset number of iterations is reached, thus obtaining the trained specific residual adapter.

[0037] In summary, during the specific residual stage, the system switches to a specific process dataset, reverses the parameter activation strategy, completely freezes the previously trained deep features, and instead activates and trains only the specific residual LoRA module (SR-LoRA) mounted in the high-resolution edge layers (Layers 1, 2, 6, 7), which is specifically used to characterize the micro-stitch topology and needle direction details.

[0038] It should be noted that the first loss function uses diffusion loss to constrain the macroscopic composition and color distribution of the generated image; The first loss function, i.e., diffusion loss The expression is:

[0039] In the formula, To inject the true common representation features, The model predicts the output; The image represents the passage through the latent space. Encoding after adding noise at each time step; General prompt words ; The second loss function adopts a joint loss of weighted summation of structural tensor loss and spectral preservation loss. The structural tensor loss uses the spatial domain edge orientation angle to constrain the topological orientation of micro-traces, and the spectral preservation loss aligns high-frequency energy on the Fourier logarithmic amplitude spectrum to restore the high reflective gloss texture of the silk fibers. The structural tensor loss The expression is:

[0040] In the formula, the structure tensor is used. Extract the covariance matrix of the local gradient of the image, and generate an image by comparison. Compared with real samples The difference in the direction field forces the model to learn specific needle operation logic.

[0041] The spectrum preservation loss The expression is:

[0042] In the formula, Represents the amplitude spectrum. It can "energy-inject" the edge details of embroidery threads, enabling shallow network parameters to focus on the most drastically changing subtle textures in the image, thereby restoring the distinct physical texture of each thread and ensuring that the generated embroidery works maintain extremely high clarity and sharpness even under magnification.

[0043] In addition to the above-mentioned solutions, this embodiment also provides a detailed description of the significant technical advancements of this application through experimental comparison data: The experiment was conducted on the Qinghai embroidery dataset constructed in this patent. In order to accurately evaluate the visual realism and cultural conformity of the generated images in the specific field of Qinghai embroidery, this embodiment adopted a targeted evaluation scheme: FID (Domain-specific Fréchet Inception Distance): This value is used to accurately measure the similarity between the generated image and the real Qinghai embroidery in terms of macroscopic visual features, material aesthetics, and global distribution. The value is calculated by fine-tuning the feature extraction network on the Qinghai embroidery dataset. The lower the value, the closer the generated embroidery global image is to the distribution of the real sample. LPIPS (Learned Perceptual Image Patch Similarity): This metric quantifies the microscopic distortion of generated images in depth perception space. It closely matches the human visual system's perception of local edges and texture details in embroidery images; a lower value indicates higher visual fidelity at the microscopic perception level. OCS (Orientation Coherence Score): Based on the inherent physical principles of embroidery texture, it extracts the spatial domain directional coherence of the image and monitors in real time whether the stitches have topological breaks or isotropic blurring. The higher the value, the more the generated stitch direction conforms to the needlework technique of real Qing embroidery. HER (High-frequency Energy Ratio): Calculated using Fourier transform or spectrum preservation, this value accurately measures the high-frequency energy distribution of microfibers in the frequency domain. The higher the value, the better the generated image reproduces the unique high-frequency texture and micro-luster of the Qingxiu silk thread.

[0044] Example 2: Figure 3 This is a schematic diagram of a diffusion model embroidery generation apparatus based on aesthetic and technological decoupling, according to another exemplary embodiment. The apparatus includes: Text prompt word acquisition module 1: used to acquire text prompt words, which contain descriptive information about the target embroidery image; Text encoding module 2: used to input the text prompt words into the text encoder for encoding, and obtain the text conditional feature vector; Image generation module 3: used to input the text conditional feature vector into the diffusion model, and generate the target embroidery image through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image.

[0045] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0046] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0047] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0048] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0049] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0050] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0051] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0052] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0053] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An embroidery generation method based on a diffusion model decoupling aesthetics and craftsmanship, characterized in that, The method includes: Obtain text prompts, which contain descriptive information about the target embroidery image; The text prompt words are input into a text encoder for encoding to obtain a text conditional feature vector; The text conditional feature vector is input into the diffusion model, and the target embroidery image is generated through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image.

2. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship as described in claim 1, characterized in that, The training of the common representation adapter and the specific residual adapter includes: Obtain both a general embroidery style dataset and a specific process dataset; Obtain the true common characteristics of the general embroidery style dataset, and obtain the true microscopic process-specific characteristics of the specific process dataset; Obtain the first text prompt word of the general embroidery style dataset, and obtain the second text prompt word of the specific process dataset; Activate the common representation adapter of the deep bottleneck layer of the backbone network and freeze the specific residual adapter of the shallow high-resolution layer of the backbone network. The first text prompt word is input into the backbone network, and the predicted common representation features are output through the activated common representation adapter. The error between the predicted common representation features and the real common representation features is measured by a preset first loss function. The common representation adapter is iteratively trained until a preset iteration stopping condition is met, and a trained common representation adapter is obtained. Activate the specific residual adapter of the shallow high-resolution layer of the backbone network and freeze the trained common representation adapter; The second text prompt is input into the backbone network, and the predicted micro-process specific features are output through the activated specific residual adapter. The error between the predicted micro-process specific features and the actual micro-process specific features is measured by a preset second loss function. The specific residual adapter is iteratively trained until the preset iteration stopping condition is met, and the trained specific residual adapter is obtained.

3. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship as described in claim 2, characterized in that, The acquisition of the general embroidery style dataset and the specific process dataset includes: A multimodal embroidery dataset is obtained, and the multimodal data is decoupled from the multimodal embroidery dataset. Images containing diverse cross-style embroidery images are assigned to the general embroidery style dataset, and images of embroidery images with specific needlework techniques are assigned to the specific technique dataset.

4. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship as described in claim 2, characterized in that, The first loss function is the diffusion loss function; The second loss function is a joint loss function obtained by weighted summation of the structure tensor loss function and the spectrum preservation loss function.

5. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship as described in claim 1, characterized in that, After obtaining the target embroidery image, the process further includes: The target embroidery image output by the decoder is input into the super-resolution reconstruction module, which performs a resolution magnification operation on the target embroidery image to obtain an ultra-high-definition embroidery image.

6. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship as described in claim 5, characterized in that, After obtaining the ultra-high-definition embroidery image, the process also includes: The ultra-high-definition embroidery image is converted from the RGB color gamut to the industrial-grade CMYK color gamut or the Pantone special color gamut for intangible cultural heritage, and then subjected to noise reduction and contrast optimization processing to obtain an industrial-grade lossless embroidery image.

7. The diffusion model embroidery generation method based on the decoupling of aesthetics and craftsmanship according to claim 1, characterized in that, The backbone network adopts the U-Net network, the deep bottleneck layer is Layers 3, 4, and 5 of the U-Net network, and the shallow high-resolution layer is Layers 1, 2, 6, and 7 of the U-Net network.

8. An embroidery generation device based on the decoupling of aesthetics and craftsmanship using a diffusion model, characterized in that, The device includes: Text prompt word acquisition module: used to acquire text prompt words, which contain descriptive information about the target embroidery image; Text encoding module: used to input the text prompt words into the text encoder for encoding, and obtain the text conditional feature vector; Image generation module: used to input the text conditional feature vector into the diffusion model, and generate the target embroidery image through the diffusion model; The diffusion model includes a backbone network, a feature fusion module, and a decoder; the output of the backbone network is connected to the feature fusion module, and the output of the feature fusion module is connected to the decoder. The backbone network includes a deep bottleneck layer and a shallow high-resolution layer. The deep bottleneck layer contains a pre-trained common representation adapter, and the shallow high-resolution layer contains a pre-trained specific residual adapter. The common representation adapter is used to generate common representation features based on the input text conditional feature vector, and the specific residual adapter is used to generate micro-process specific features based on the input text conditional feature vector. The feature fusion module is used to perform high-order aggregation and fusion of the received common characterization features and micro-process-specific features in the manifold space to generate high-order fused features. The decoder is used to render and parse the latent spatial representation of the received high-order fusion features into an embroidery raster image based on the received high-order fusion features, thereby obtaining the target embroidery image.

Images