Text-driven Knitted Product Image Generation and Editing Method and Device

Abstract

The present invention discloses a text-driven knitted product image generation and editing method and device, relating to the field of computer graphics. The method comprises: S1, obtaining knitted product text, an editing mask, and an input image, and splicing the editing mask and a reference image to obtain a spliced image; S2, constructing and training a text-driven knitted product image generation and editing model comprising a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network to obtain a trained knitted product image generation and editing model; S3, inputting the knitted product text, the editing mask, and the reference image into the trained text-driven knitted product image generation and editing model to obtain a knitted product image. The text-driven knitted product image generation and editing model constructed by the present invention integrates knitted product image generation and editing functions, simplifies the model, and achieves deep fusion of multimodal features and precise understanding and execution of editing instructions.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a text-driven knitted product image generation and editing method and device. Background Art

[0002] With the accelerating digital transformation of the textile and fashion industries, traditional knitwear product design can no longer meet the personalized needs of consumers. Traditional knitwear product design processes typically rely on manual drawing or specialized design software, which is not only time-consuming and labor-intensive but also requires high levels of professional skills from designers, limiting the speed and scope of design innovation. Furthermore, in real-world applications, consumers want to generate new knitwear product images based on text descriptions or modify existing images (such as changing colors or adjusting textures). However, existing image generation or editing methods mostly focus on optimizing a single task and lack a unified framework to support both image generation and editing.

[0003] Therefore, exploring an efficient and accurate unified framework for knitted product image generation and editing to meet the market demand for personalized design and rapid iteration has become a key issue that needs to be urgently addressed in the current textile and fashion industry. Summary of the Invention

[0004] In response to the above problems, the present invention proposes a text-driven knitted product image generation and editing method and device. By integrating the knitted product image generation and editing tasks into one model, the complexity of model deployment is simplified, and the generalization ability and ease of use of the model are improved; a visual language model encoder is used to perform high-dimensional semantic joint modeling of knitted product text, input image and editing mask, thereby realizing deep fusion of multimodal features; and an image diffusion module, multi-level residual upsampling and multi-level residual downsampling are used to achieve accurate understanding and execution of editing instructions without destroying the overall structure and details of the image.

[0005] On the one hand, the text-driven knitted product image generation and editing method has the following specific steps:

[0006] S1, obtaining knitted product text, an edit mask, and an input image, and splicing the edit mask and the input image to obtain a spliced image;

[0007] S2, constructing and training a text-driven knitted product image generation and editing model including a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network to obtain a trained knitted product image generation and editing model;

[0008] The multimodal fusion feature network includes a visual language model encoder, a variational autoencoder, and a multi-level residual downsampling module; the input knitted product text and the spliced image are fused by the visual language model encoder to obtain an initial multimodal feature; the input image is encoded by the variational autoencoder and Gaussian noise embedded to obtain an initial latent feature, and the initial latent feature is subjected to multi-scale feature fusion by the multi-level residual downsampling module to obtain a multi-level initial latent feature; the multimodal feature and the multi-level initial latent feature are spliced along the channel dimension to obtain a multimodal fusion feature and output;

[0009] The multi-layer denoising network includes several denoising network layers, each of which includes a first image diffusion module and a second image diffusion module; the first layer of the multi-layer denoising network takes the multimodal fusion feature as input, the remaining layers take the output of the previous layer as input, and the last layer outputs the final denoised multimodal fusion feature to the knitted image reconstruction network; the first image diffusion module denoises the input to obtain a preliminary denoised multimodal fusion feature, and the preliminary denoised multimodal fusion feature is segmented along the channel dimension to obtain an intermediate multimodal feature and an intermediate latent feature; the intermediate multimodal feature and the intermediate latent feature after time step embedding are spliced along the channel dimension, and then denoised by the second image diffusion module, and the denoised multimodal fusion feature is output;

[0010] The knitted image reconstruction network segments the final denoised multimodal fusion features to obtain denoised latent features; the denoised latent features are input into a multi-level residual upsampling module to reconstruct the knitted image and output a knitted product image;

[0011] S3, input the knitted product text, editing mask and input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image.

[0012] Preferably, the text-driven knitted product image generation and editing model adopts time step embedding, which acts on the initial latent features, multi-level residual downsampling module, first image diffusion module, intermediate latent features, second image diffusion module, denoising latent features and multi-level residual upsampling module respectively.

[0013] Preferably, the time step embedding is expressed as:

[0014] ;

[0015] in, represents the time step embedding, Represents the features after time step embedding; Represents the features to be embedded; represents the time step noise attenuation coefficient.

[0016] Preferably, the time step noise attenuation coefficient is expressed as:

[0017] ;

[0018] in, Indicates the current time step.

[0019] Preferably, the Gaussian noise embedding is expressed as:

[0020] ;

[0021] in, Represents the features after Gaussian noise embedding; Represents the features to be embedded; represents standard Gaussian noise; represents the Gaussian noise attenuation coefficient.

[0022] Preferably, the Gaussian noise attenuation coefficient is expressed as:

[0023] ;

[0024] Among them, T represents the total step length and t represents the current time step.

[0025] Preferably, the multi-level residual downsampling module is expressed as:

[0026] ;

[0027] in, represents the initial latent features after enhancement; Indicates an operation repeated N times; Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 2; Represents the ReLU activation function; Represents normalization operation; Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 1; represents the initial latent features.

[0028] Preferably, the multi-level residual upsampling module is expressed as:

[0029] ;

[0030] in, represents the enhanced denoised latent features; Indicates an operation repeated N times; Indicates a transposed convolution operation with a convolution kernel of 3×3 and a stride of 2; Represents the ReLU activation function; Represents normalization operation; Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 1; represents the denoised latent features.

[0031] Preferably, the first image diffusion module and the second image diffusion module have the same structure, and their respective processing processes are specifically as follows:

[0032] First, the multimodal fusion features of the input image diffusion module are input into the multi-head self-attention layer to obtain the enhanced multimodal fusion features after the multi-head self-attention layer, which is expressed as:

[0033] ;

[0034] in, Represents the initial multimodal fusion features; Represents the enhanced multimodal fusion features after the multi-head self-attention layer; Represents a splicing operation; represents a multi-head attention layer; Represents the ReLU activation function; Represents normalization operation; represents the time step embedding;

[0035] Then, the enhanced multimodal fusion features after the multi-head self-attention layer are input into the forward feedback layer, and the enhanced multimodal fusion features after the forward feedback layer are output; it can be expressed as:

[0036] ;

[0037] in, represents the enhanced multimodal fusion features after the forward feedback layer, represents the feedforward layer.

[0038] On the other hand, the text-driven knitted product image generation and editing device includes the following:

[0039] a knitted product data acquisition module, configured to acquire knitted product text, an edit mask, and an input image, and to splice the knitted product edit mask and the input image to obtain a spliced image;

[0040] A model building and training module is used to build and train a text-driven knitted product image generation and editing model that includes a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network, thereby obtaining a trained knitted product image generation and editing model.

[0041] The multimodal fusion feature network includes a visual language model encoder, a variational autoencoder, and a multi-level residual downsampling module; the input knitted product text and the spliced image are fused by the visual language model encoder to obtain an initial multimodal feature; the input image is encoded by the variational autoencoder and Gaussian noise embedded to obtain an initial latent feature, and the initial latent feature is subjected to multi-scale feature fusion by the multi-level residual downsampling module to obtain a multi-level initial latent feature; the multimodal feature and the multi-level initial latent feature are spliced along the channel dimension to obtain a multimodal fusion feature and output;

[0042] The multi-layer denoising network includes several denoising network layers, each of which includes a first image diffusion module and a second image diffusion module; the first layer of the multi-layer denoising network takes the multimodal fusion feature as input, the remaining layers take the output of the previous layer as input, and the last layer outputs the final denoised multimodal fusion feature to the knitted image reconstruction network; the first image diffusion module denoises the input to obtain a preliminary denoised multimodal fusion feature, and the preliminary denoised multimodal fusion feature is segmented along the channel dimension to obtain an intermediate multimodal feature and an intermediate latent feature; the intermediate multimodal feature and the intermediate latent feature after time step embedding are spliced along the channel dimension, and then denoised by the second image diffusion module, and the denoised multimodal fusion feature is output;

[0043] The knitted image reconstruction network segments the final denoised multimodal fusion features to obtain denoised latent features; the denoised latent features are input into a multi-level residual upsampling module to reconstruct the knitted image and output a knitted product image;

[0044] The text-driven knitted product image generation and editing model adopts time-step embedding, which acts on the initial latent features, multi-level residual downsampling module, first image diffusion module, intermediate latent features, second image diffusion module, denoising latent features and multi-level residual upsampling module respectively;

[0045] The knitted product image generation module inputs the knitted product text, editing mask and input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) This paper constructs a unified knitted product image generation and editing framework, integrating image generation and editing tasks into one model. This not only simplifies the complexity of model deployment, but also significantly improves the model's generalization ability and ease of use, greatly improving work efficiency and user experience.

[0048] (2) The present invention uses a visual language model encoder to perform high-dimensional semantic joint modeling of knitted product text, input image, and edit mask, achieving deep fusion of multimodal features; this fusion approach enables the model to more accurately understand the user's intention and generate knitted product images that are more consistent with the text description;

[0049] (3) The present invention, through the design of an image diffusion module and a multi-level residual network including multi-level residual upsampling and multi-level residual downsampling, can achieve accurate understanding and execution of editing instructions without destroying the overall structure and details of the image, such as changing the color of the knitted product image and adjusting the texture, which is of great significance for meeting the personalized needs of consumers. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be described in further detail below with reference to the accompanying drawings;

[0051] Figure 1 Flowchart of a text-driven knitted product image generation and editing method according to an embodiment of the present invention;

[0052] Figure 2 A schematic structural diagram of a knitted product image generation and editing model of a text-driven knitted product image generation and editing method according to an embodiment of the present invention;

[0053] Figure 3 This is a structural diagram of an image diffusion module of a text-driven knitted product image generation and editing method according to an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of knitted product image generation and editing functions of a text-driven knitted product image generation and editing method according to an embodiment of the present invention;

[0055] Figure 5 This is a structural block diagram of a text-driven knitted product image generation and editing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below through specific embodiments.

[0057] like Figure 1 As shown in FIG, the text-driven knitted product image generation and editing method includes the following specific steps:

[0058] S1, obtaining a knitted product text, an edit mask, and an input image, and splicing the knitted product edit mask and the input image to obtain a spliced image.

[0059] Knitted product text refers to a natural language description used to describe the characteristics, style, color, texture, and style of knitted product images; the image is a reference image or a source image to be edited; the mask is a matrix of the same size as the input image, which serves the editing task and is used to limit the editing area. The element value in the matrix is 0 or 1, and 1 indicates the editing area.

[0060] S2, constructs a text-driven knitted product image generation and editing model including a multimodal fusion feature network, a multi-layer denoising network and a knitted image reconstruction network and trains it to obtain a trained knitted product image generation and editing model.

[0061] See also Figure 2 As shown in the figure, the knitted product image generation and editing model includes a multimodal feature fusion network, a multi-layer denoising network and a knitted image reconstruction network, which are used for high-dimensional semantic joint modeling, layer-by-layer denoising processing and knitted image reconstruction, respectively; the multimodal feature fusion network includes a vision language model encoder (VLM, or VLM encoder), initial multimodal features, a variational auto-encoder (VAE, or VAE encoder), initial latent features and a multi-level residual downsampling module; the multi-layer denoising network includes an image diffusion module (Diffusion Image Transformer, DIT, or DIT module), intermediate multimodal features and intermediate latent features. The multi-layer denoising network includes several layers, each layer includes two diffusion modules; the knitted image reconstruction network includes denoising multimodal features, denoising latent features, a multi-level residual upsampling module and knitted image output.

[0062] The multimodal feature fusion network receives text input, image input, and mask input (i.e., knitted product text, edit mask, and input image) and performs high-dimensional semantic joint modeling of these inputs through a visual language model encoder to obtain initial multimodal features. A variational autoencoder is used to encode the image and embed Gaussian noise to obtain initial latent features. A multi-level residual downsampling module is used to perform multi-scale feature fusion on the initial latent features to obtain multi-level initial latent features. The initial multimodal features and multi-level initial latent features are then concatenated along the channel dimension to form multimodal fusion features. A multi-layer denoising network receives the multimodal fusion features and performs layer-by-layer denoising on them using an image diffusion module. The image diffusion module, consisting of a multi-head self-attention layer and a forward feedback layer, achieves fine-grained denoising of the multimodal fusion features through time-step embedding. After multiple denoising steps, denoised multimodal features and denoised latent features are obtained. The knitted image reconstruction network receives the denoised latent features and gradually restores image details through a multi-level residual upsampling module, ultimately outputting a high-resolution knitted product image.

[0063] In a specific embodiment, the visual language model encoder uses the existing Qwen-VL 7B model, supports multi-resolution image input (256×256 to 1024×1024), and can better perform fine-grained semantic analysis of knitted products (such as "change the round neck to a V-neck"); the variational autoencoder adopts the existing FLUX-schnell architecture, and the number of latent space channels is expanded to 128 to retain knitted texture details (such as threads, cables, etc.); the image diffusion module uses 12 layers, supports the input of long text instructions, and can converge within a shorter inference step.

[0064] In a specific embodiment, during the variational autoencoder encoding stage, time-dependent Gaussian noise is injected into the potential representation of the input image I. The formula for embedding Gaussian noise is as follows:

[0065] ;

[0066] in, Represents the features after Gaussian noise embedding of X; Represents the features to be embedded; represents standard Gaussian noise, , the dimension is consistent with the latent space (default 256×256); represents the Gaussian noise attenuation coefficient, following the cosine schedule, , T represents the total step length, and t represents the current time step.

[0067] In a specific embodiment, knitted product images have overall structural stability and coherence, as well as complex local textures, which determine the task characteristics of their overall to local generation process. Therefore, time step embedding adopts a phased dynamic scheduling strategy based on the task characteristics of knitted product generation. The formula for embedding time step noise is expressed as:

[0068] ;

[0069] in, Represents the features after time step embedding of X; represents the time step noise attenuation; X represents the feature to be embedded.

[0070] Specifically, time step embedding achieves progressive embedding of noise through cosine scheduling. , retain low-intensity basic noise, ensure the high-frequency texture of the input image (such as thread, twist, etc.), and smoothly transition to the mid-term stage through cosine scheduling; in the mid-term stage , using a 0.8 scaling factor to control the noise attenuation rate, the noise intensity gradually decreases according to the cosine schedule, and ensures that decays to zero; in the later stages , completely stops the noise injection, and the residual upsampling module restores the high-resolution details. The specific parameters of the time step noise attenuation are as follows:

[0071] ;

[0072] in, represents the time step noise attenuation coefficient, Represents the time step.

[0073] During the downsampling process, time-step embedding is used to adjust the intensity of noise to extract image features at different scales. The image diffusion module utilizes time-step embedding to perform layer-by-layer denoising of multimodal fusion features through a multi-head self-attention layer and a forward feedback layer. Time-step embedding is used in the diffusion module to control the denoising process, gradually reducing noise and restoring image clarity and detail. During the upsampling phase, time-step embedding is used to control the restoration of image detail. Time-step embedding ensures that noise injection is completely stopped in the later stages, allowing the residual upsampling module to effectively restore high-resolution image detail.

[0074] Visual Language Model Encoder. In a specific embodiment, the knitted product text, edit mask, and input image are input to a trained knitted product image generation and editing model. The visual language model encoder processes the input information to obtain the initial multimodal features of the input information, as follows:

[0075] The visual language model encoder is used to perform high-dimensional semantic joint modeling on the input text, image, and mask to obtain the initial multimodal features of the input information, which can be expressed as:

[0076] ;

[0077] in, Represents the initial multimodal features of the input information, represents the visual language model encoder, Indicates the input text information. Represents the input image information, Indicates the input mask information, Represents a splicing operation.

[0078] Variational Autoencoder. In a specific embodiment, an image is input into a variational autoencoder and embedded with Gaussian noise to obtain initial latent features. A multi-level residual downsampling module performs multi-scale feature fusion on the initial latent features to obtain multi-level initial latent features, as follows:

[0079] The image is encoded using a variational autoencoder and embedded with Gaussian noise to obtain the initial potential features of the input image, which can be expressed as:

[0080] ;

[0081] in, represents the initial latent features of the input image, represents Gaussian noise embedding, represents a variational autoencoder.

[0082] Multi-level residual downsampling module. The initial latent features of the input image are input to the multi-level residual downsampling module for multi-scale feature fusion to obtain the multi-level initial latent features, which can be expressed as:

[0083] ;

[0084] in, represents the multi-level initial latent features, represents the multi-level residual downsampling module, Represents the noise embedding at the current time step.

[0085] In a specific embodiment, a multi-level residual downsampling module is composed of N repeated residual convolution layers and downsampling layers. The initial latent features of the input image are input to the multi-level residual downsampling module to obtain the enhanced initial latent features of the input image, which is expressed as:

[0086] ;

[0087] in, represents the initial latent features of the enhanced input image, Represents an operation repeated N times, Represents a convolution operation with a convolution kernel of 3×3 and a stride of 2. represents the ReLU activation function, represents the normalization operation, Represents a convolution operation with a kernel size of 3×3 and a stride of 1.

[0088] In a specific embodiment, the multimodal features of the input information and the multi-level initial latent features are spliced along the channel dimension to obtain a multimodal fusion feature, which is expressed as:

[0089] ;

[0090] in, represents multimodal fusion features, Represents a splicing operation.

[0091] In a specific embodiment, the multimodal fusion features are input into the image diffusion module to obtain the preliminary denoised multimodal fusion features, which are segmented along the channel dimension to obtain intermediate multimodal features and intermediate latent features. The intermediate multimodal features and the intermediate latent features after time step embedding are spliced along the channel dimension, passed through the image diffusion module and then segmented to obtain denoised multimodal features and denoised latent features, which specifically includes:

[0092] The multimodal fusion features are input into the image diffusion module to obtain the multimodal fusion features for preliminary denoising, as shown below:

[0093] ;

[0094] in, represents the multimodal fusion features of preliminary denoising, Represents the image diffusion module.

[0095] The multimodal fusion features of the preliminary denoising are split along the channel dimension to obtain intermediate multimodal features and intermediate latent features, as shown below:

[0096] ;

[0097] in, represents the intermediate multimodal features, represents the intermediate latent feature, Represents a matrix split operation.

[0098] The intermediate multimodal features and the intermediate latent features after time step embedding are concatenated along the channel dimension, passed through the image diffusion module, and then segmented along the channel dimension to obtain the denoised multimodal features and denoised latent features, as shown below:

[0099] ;

[0100] in, represents the denoising multimodal features, represents the denoised latent features.

[0101] In a specific embodiment, reference Figure 3 ,The image diffusion module consists of a multi-head self-attention layer and a forward feedback layer, and uses time step embedding to achieve layer-by-layer denoising of multimodal fusion features, as follows:

[0102] First, the multimodal fusion features are input into the multi-head self-attention layer. The specific calculation is as follows:

[0103] ;

[0104] in, represents the initial multimodal fusion features, represents the enhanced multimodal fusion features after the multi-head self-attention layer, represents a multi-head attention layer; Represents the ReLU activation function.

[0105] Then, the enhanced multimodal fusion features are input into the forward feedback layer, and the specific calculation is as follows:

[0106] ;

[0107] in, represents the enhanced multimodal fusion features after the forward feedback layer, represents the forward feedback layer; Represents the ReLU activation function.

[0108] In a specific embodiment, the denoised latent features are input into a multi-level residual upsampling module for knitted image reconstruction, and finally a high-resolution knitted image is obtained, which is expressed as:

[0109] ;

[0110] in, represents the final high-resolution knitted image, Represents a multi-level residual upsampling module.

[0111] In a specific embodiment, a multi-level residual upsampling module is composed of N repeated residual convolution layers and upsampling layers. The denoised latent features are input into the multi-level residual upsampling module to obtain enhanced denoised latent features, as shown below:

[0112] ;

[0113] in, represents the enhanced denoised latent representation, Represents a transposed convolution operation with a kernel size of 3×3 and a stride of 2.

[0114] S3, inputting the knitted product text, the knitted product editing mask and the input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image.

[0115] like Figure 4 As shown, the knitted product image generation and editing functions are implemented as follows:

[0116] Editing functionality is achieved by combining the knitted product text, a knitted product edit mask, and an input image. Specifically, the mask specifies the area in the image to be edited, defining the scope of the edit operation. The text instructions provide specific information on how to edit that area, such as color changes or texture adjustments.

[0117] If only a text description is input without an original image, the model identifies this as an image generation task and generates a new knitted product image from scratch based on the text description. If both a text description and an original knitted product image are input, the model prefers image generation when no mask is provided. However, when a mask is provided, the model incorporates the mask information to perform image editing operations, as the presence of a mask typically indicates that specific areas of the image need to be modified.

[0118] like Figure 5 As shown, the present invention also discloses a text-driven knitted product image generation and editing device, comprising:

[0119] The knitted product data acquisition module 501 is used to acquire the knitted product text, the edit mask and the input image, and to splice the knitted product edit mask and the input image to obtain a spliced image;

[0120] A model construction and training module 502 is used to construct and train a text-driven knitted product image generation and editing model including a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network to obtain a trained knitted product image generation and editing model;

[0121] The multimodal fusion feature network includes a visual language model encoder, a variational autoencoder, and a multi-level residual downsampling module; the input knitted product text and the spliced image are fused by the visual language model encoder to obtain an initial multimodal feature; the input image is encoded by the variational autoencoder and Gaussian noise embedded to obtain an initial latent feature, and the initial latent feature is subjected to multi-scale feature fusion by the multi-level residual downsampling module to obtain a multi-level initial latent feature; the multimodal feature and the multi-level initial latent feature are spliced along the channel dimension to obtain a multimodal fusion feature and output;

[0122] The multi-layer denoising network includes several denoising network layers, each of which includes a first image diffusion module and a second image diffusion module; the first layer of the multi-layer denoising network takes the multimodal fusion feature as input, the remaining layers take the output of the previous layer as input, and the last layer outputs the final denoised multimodal fusion feature to the knitted image reconstruction network; the first image diffusion module denoises the input to obtain a preliminary denoised multimodal fusion feature, and the preliminary denoised multimodal fusion feature is segmented along the channel dimension to obtain an intermediate multimodal feature and an intermediate latent feature; the intermediate multimodal feature and the intermediate latent feature after time step embedding are spliced along the channel dimension, and then denoised by the second image diffusion module, and the denoised multimodal fusion feature is output;

[0123] The knitted image reconstruction network segments the final denoised multimodal fusion features to obtain denoised latent features; the denoised latent features are input into a multi-level residual upsampling module to reconstruct the knitted image and output a knitted product image;

[0124] The knitted product image generation module 503 inputs the knitted product text, the editing mask and the input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image.

[0125] The specific implementation of the text-driven knitted product image generation and editing device is the same as the text-driven knitted product image generation and editing method, and will not be repeated in this embodiment.

[0126] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A text-driven knitted product image generation and editing method, characterized in that: The steps include: S1, obtaining knitted product text, an edit mask, and an input image, and splicing the edit mask and the input image to obtain a spliced image; S2, constructing and training a text-driven knitted product image generation and editing model including a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network to obtain a trained knitted product image generation and editing model; The multimodal fusion feature network includes a visual language model encoder, a variational autoencoder, and a multi-level residual downsampling module; the input knitted product text and spliced image are fused through the visual language model encoder to obtain initial multimodal features; The input image is encoded by a variational autoencoder and Gaussian noise is embedded to obtain the initial latent features. The initial latent features are then fused with multi-scale features by a multi-level residual downsampling module to obtain multi-level initial latent features. Multimodal features and multi-level initial latent features are spliced along the channel dimension to obtain multimodal fusion features and output; The multi-layer denoising network includes several denoising network layers, each of which includes a first image diffusion module and a second image diffusion module; the first layer of the multi-layer denoising network takes the multimodal fusion feature as input, the remaining layers take the output of the previous layer as input, and the last layer outputs the final denoising multimodal fusion feature to the knitted image reconstruction network; The first image diffusion module denoises the input to obtain a preliminary denoised multimodal fusion feature, and then segments the preliminary denoised multimodal fusion feature along the channel dimension to obtain an intermediate multimodal feature and an intermediate latent feature. The intermediate multimodal features and the intermediate latent features after time step embedding are spliced along the channel dimension, and then denoised by the second image diffusion module to output the denoised multimodal fusion features; The knitted image reconstruction network segments the final denoised multimodal fusion features to obtain denoised latent features; the denoised latent features are input into a multi-level residual upsampling module to reconstruct the knitted image and output a knitted product image; S3, inputting the knitted product text, the editing mask, and the input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image; The multi-level residual downsampling module is expressed as: in, represents the initial latent features after enhancement; [·] ×N Indicates N repeated operations; Conv (3·3,2) Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 2; ReLU activation function; Norm(·) represents the normalization operation; Conv (3×3,1) Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 1; represents the initial latent features; The multi-level residual upsampling module is expressed as: in, represents the enhanced denoised latent features; [·] ×N Represents N repeated operations; TConv (3×3,2) Indicates a transposed convolution operation with a convolution kernel of 3×3 and a stride of 2; ReLU activation function; Norm(·) represents the normalization operation; Conv (3×3,1) Indicates a convolution operation with a convolution kernel of 3×3 and a stride of 1; represents the denoised latent features.

2. The text-driven knitting product image generation and editing method according to claim 1, characterized in that: The text-driven knitted product image generation and editing model adopts time-step embedding, which acts on the initial latent features, multi-level residual downsampling module, first image diffusion module, intermediate latent features, second image diffusion module, denoising latent features and multi-level residual upsampling module respectively.

3. The text-driven knitting product image generation and editing method according to claim 2, characterized in that: The time step embedding is represented as: Among them, Emb τ (·) represents the time step embedding, Emb τ (X) represents the feature after time step embedding; X represents the feature to be embedded; α τ represents the time step noise attenuation coefficient.

4. The text-driven knitting product image generation and editing method according to claim 3, characterized in that: The time step noise attenuation coefficient is expressed as: Where t represents the current time step.

5. The text-driven knitting product image generation and editing method according to claim 2, characterized in that: The Gaussian noise embedding is expressed as: in, represents the feature after Gaussian noise embedding; X represents the feature to be embedded; ε represents standard Gaussian noise; represents the Gaussian noise attenuation coefficient.

6. The text-driven knitting product image generation and editing method according to claim 5, characterized in that: The Gaussian noise attenuation coefficient is expressed as: Among them, T represents the total step length and t represents the current time step.

7. The text-driven knitting product image generation and editing method according to claim 2, characterized in that: The first image diffusion module and the second image diffusion module have the same structure, and their respective processing processes are specifically as follows: First, the multimodal fusion features of the input image diffusion module are input into the multi-head self-attention layer to obtain the enhanced multimodal fusion features after the multi-head self-attention layer, which is expressed as: in, Represents the initial multimodal fusion features; Represents the enhanced multimodal fusion features after the multi-head self-attention layer; represents the concatenation operation; MHSA(·) represents the multi-head attention layer; ReLU activation function; Norm(·) represents the normalization operation; Emb τ (·) represents the time step embedding; Then, the enhanced multimodal fusion features after the multi-head self-attention layer are input into the forward feedback layer, and the enhanced multimodal fusion features after the forward feedback layer are output; it can be expressed as: in, represents the enhanced multimodal fusion features after the forward feedback layer, and FF(·) represents the forward feedback layer.

8. A text-driven knitted product image generation and editing device using the text-driven knitted product image generation and editing method according to any one of claims 1 to 7, comprising: a knitted product data acquisition module, configured to acquire knitted product text, an edit mask, and an input image, and to splice the knitted product edit mask and the input image to obtain a spliced image; A model building and training module is used to build and train a text-driven knitted product image generation and editing model that includes a multimodal fusion feature network, a multi-layer denoising network, and a knitted image reconstruction network, thereby obtaining a trained knitted product image generation and editing model. The multimodal fusion feature network includes a visual language model encoder, a variational autoencoder, and a multi-level residual downsampling module; the input knitted product text and spliced image are fused through the visual language model encoder to obtain initial multimodal features; The input image is encoded by a variational autoencoder and Gaussian noise is embedded to obtain the initial latent features. The initial latent features are then fused with multi-scale features by a multi-level residual downsampling module to obtain multi-level initial latent features. Multimodal features and multi-level initial latent features are spliced along the channel dimension to obtain multimodal fusion features and output; The multi-layer denoising network includes several denoising network layers, each of which includes a first image diffusion module and a second image diffusion module; the first layer of the multi-layer denoising network takes the multimodal fusion feature as input, the remaining layers take the output of the previous layer as input, and the last layer outputs the final denoising multimodal fusion feature to the knitted image reconstruction network; The first image diffusion module denoises the input to obtain a preliminary denoised multimodal fusion feature, and then segments the preliminary denoised multimodal fusion feature along the channel dimension to obtain an intermediate multimodal feature and an intermediate latent feature. The intermediate multimodal features and the intermediate latent features after time step embedding are spliced along the channel dimension, and then denoised by the second image diffusion module to output the denoised multimodal fusion features; The knitted image reconstruction network segments the final denoised multimodal fusion features to obtain denoised latent features; the denoised latent features are input into a multi-level residual upsampling module to reconstruct the knitted image and output a knitted product image; The knitted product image generation module inputs the knitted product text, editing mask and input image into the trained text-driven knitted product image generation and editing model to obtain the knitted product image.

Images