A sketch-driven artistic image generation method, system and medium

Abstract

This invention relates to a sketch-driven art image generation method, system, and medium. The method includes the following steps: acquiring a user-provided hand-drawn sketch, text prompts, and reference art images; preprocessing the hand-drawn sketch to obtain a binary sketch image; inputting the binary sketch image, text prompts, and reference art images into a pre-trained art image generation network model, performing an iterative denoising diffusion process in the latent space to obtain the target art image; wherein the art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. Compared with existing technologies, this invention significantly improves the image quality and controllability of sketch-driven art image generation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a sketch-driven artistic image generation method, system, and medium. Background Technology

[0002] With the development of generative artificial intelligence technology, image synthesis methods based on generative models have made significant progress in tasks such as text-to-image generation, image editing, and style transfer. Among them, the Latent Diffusion Model (LDM) has become the mainstream technical solution in the field of image generation by performing a stepwise denoising process in a low-dimensional latent space, which balances generation quality and computational efficiency.

[0003] In sketch-driven image generation tasks, users typically provide hand-drawn sketches as generation criteria, aiming to generate target images that are consistent with the sketch content and have high visual quality. Compared to text prompts, hand-drawn sketches can more directly provide the shape outline, spatial layout, and local structural relationships of the target object, thus having significant application value in creative design, digital content creation, and intelligent drawing assistance scenarios.

[0004] However, hand-drawn sketches by ordinary users typically only contain rough outlines, with abstract semantic expressions and sparse information, often exhibiting problems such as missing parts, scale deviations, broken lines, and unclear structures. When such sketches are directly used as conditional input to pre-train a diffusion model, the model tends to follow the image distribution it has learned from large-scale natural image data, thus weakening the sketch's ability to constrain the generated structure. This leads to problems such as contour shifts, local structural distortions, target misalignment, or inconsistencies between the generated result and the overall content.

[0005] Some existing methods enhance the responsiveness of diffusion models to structural signals such as sketches, edge maps, or depth maps by introducing control branches, adapter modules, or additional conditional networks. However, these methods are mostly applicable to edge maps with explicit semantic features. When applied to semantically sparse hand-drawn sketches, the model struggles to accurately understand the user's intent, leading to a weakening of structural constraints. For example, patent application CN121767490A discloses a method and apparatus for generating architectural design images oriented towards architectural styles. This method extracts features from idealized, complete structural sketches through a control network and combines them with semantic conditions encoded by abstract text descriptions. It then uses a fixed fusion strategy to guide image generation within a model distribution trained on high-quality labeled data. This method can only achieve a rough coordination of structure and style under ideal input and fixed control strength, but it cannot effectively handle semantically sparse sketches to ensure structural fidelity, nor can it flexibly adjust control weights to achieve a dynamic balance between constraints and artistic freedom. On the other hand, excessively strengthening sketch constraints will compress the generative freedom of the diffusion model, limiting artistic style expression and reducing the diversity and expressiveness of the generated results.

[0006] Furthermore, existing methods typically use the overall features of a reference art image as a conditional input when performing art style transfer. This causes the content structure information in the reference image to interfere with the target generated image, resulting in a structure inconsistency between the target image and the sketch. Therefore, how to achieve coordinated control of text semantic completion and reference image style transfer while maintaining the fidelity of the sketch structure, and how to strike a balance between structural constraints and generative freedom, has become a pressing technical problem in this field. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a sketch-driven artistic image generation method, system and medium, which significantly improves the image quality and controllability of sketch-driven artistic image generation.

[0008] The objective of this invention can be achieved through the following technical solutions: A sketch-driven method for generating artistic images includes the following steps: Collect hand-drawn sketches, text prompts, and reference art images provided by users; The hand-drawn sketch is preprocessed to obtain a binary sketch image; The binary sketch image, text prompt, and reference art image are input into a pre-trained art image generation network model, and an iterative denoising diffusion process is performed in the latent space to obtain the target art image. The art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from the binary sketch image and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of the reference art image. The content-style adaptive fusion module fuses the text embeddings and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising diffusion process based on the multi-scale structure mask and the fused text and style embeddings in the latent space, and gradually recovers the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

[0009] Furthermore, the text prompts are used to supplement semantic details not explicitly expressed in the hand-drawn sketches, including object categories, scene descriptions, and other supplementary semantic information. The reference art image is used to provide references for the target art image in terms of artistic style, color distribution, texture features, and brushstroke expression.

[0010] Furthermore, the image preprocessing includes image scaling, edge enhancement, and binarization. The binary sketch image is used to characterize the contour structure and spatial layout information of the target art image, and the structure of the target art image is consistent with the structure of the hand-drawn sketch.

[0011] Further, the specific steps of inputting the binary sketch image, text prompt, and reference artistic image into a pre-trained artistic image generation network model, and performing an iterative denoising diffusion process in the latent space to obtain the target artistic image include: Extract the text embedding of the text prompt using a text encoder; Visual features of the reference art image are extracted using an image encoder; The binary sketch image is downsampled by a factor of the feature map resolution of different network layers in the denoising U-Net network through the dual mask structure injection module, generating a multi-scale structure mask corresponding to the feature map size of each network layer, and then injected into the denoising U-Net network. The visual features of the reference art image are decoupled by a query transformer to extract the style embedding. The text embedding and style embedding are injected into the denoising U-Net network using a key-value fusion strategy through the content style adaptive fusion module; The denoising U-Net network predicts noise during the diffusion process of iterative denoising based on random noise, multi-scale structure mask, text embedding, and style embedding, and gradually recovers the target latent variables corresponding to the target artistic image. The target latent variables are decoded using a VAE decoder to obtain the target artistic image.

[0012] Furthermore, the multi-scale structure mask is: In the formula, For the first Multi-scale structure mask of layered network layers, For the first The height of the feature maps of the layer network layers, For the first The width of the feature map of a layer in a network; The specific steps for injecting the multi-scale structure mask into the denoising U-Net network include: The potential diffusion features of the current network layer of the denoised U-Net network and its corresponding multi-scale structure mask are calculated element-wise to obtain edge region features and non-edge region features. Independent trainable channel scale parameters and bias parameters are set for the edge region features and non-edge region features respectively, and the edge region features and non-edge region features are adaptively adjusted. After adaptively adjusting the edge region features and non-edge region features, they are fused element-wise and then used as the output features of the current network layer to be input into subsequent network layers.

[0013] Furthermore, the specific steps of injecting the text embedding and style embedding into the denoising U-Net network using a key-value fusion strategy through the content-style adaptive fusion module include: The text embedding and style embedding are mapped using independent learnable linear projection matrices to obtain their respective key and value representations; Style embedding is used as a supplement to text embedding, and the two are concatenated at the token dimension to obtain the fused key and value representations. In each cross-attention layer of the denoising U-Net network, the feature map of the current layer is flattened and used as a query. Cross-attention calculation is then performed with the fused key and value representations, thereby injecting text content and style information into the diffusion process of the denoising U-Net network.

[0014] Furthermore, the specific steps of using the denoising U-Net network to predict noise and gradually recover the target latent variables corresponding to the target artistic image during the iterative denoising diffusion process based on random noise, multi-scale structural masks, text embeddings, and style embeddings include: A random noise is generated by sampling in the potential space as the starting point for the diffusion process; The noise-added latent features at the current time step are input into the denoising U-Net network, and the noise components contained in the noise-added latent features at the current time step are predicted by combining multi-scale structure mask, text embedding and style embedding. The noise latent features at the current time step are updated based on the noise components, and the updated noise latent features are used as the input for the next round of denoising. When the iterative denoising loop reaches time step 0, the diffusion process terminates, and the target latent variable corresponding to the target artistic image is obtained.

[0015] Furthermore, the artistic image generation network model is trained using a diffusion reconstruction loss function, which is: In the formula, For the diffusion reconstruction loss function, For mathematical expectation, As a potential feature, It is Gaussian noise. For reference to the visual characteristics of artistic images, For multi-scale structure masks, The noise components predicted by the denoising U-Net network. For time steps The potential features of noise addition For text prompts, For time steps; The latent features are obtained by encoding the training images into the latent space using a VAE encoder.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, enables the sketch-driven artistic image generation method described above.

[0017] According to another aspect of the present invention, a sketch-driven artistic image generation system is provided, comprising: The data acquisition module is used to collect hand-drawn sketches, text prompts, and reference art images provided by the user; The sketch processing module is used to perform image preprocessing on the hand-drawn sketch to obtain a binary sketch image; The art image generation module is used to input the binary sketch image, text prompt, and reference art image into a pre-trained art image generation network model, and perform an iterative denoising diffusion process in the latent space to obtain the target art image; The art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from the binary sketch image and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of the reference art image. The content-style adaptive fusion module fuses the text embeddings and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising diffusion process based on the multi-scale structure mask and the fused text and style embeddings in the latent space, and gradually recovers the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention designs an art image generation network model that includes a dual-mask structure injection module, a query transformer, and a content style adaptive fusion module. It simultaneously inputs hand-drawn sketches, text prompts, and reference art images into a pre-trained art image generation network model to generate the target art image. This solves the technical bottleneck of existing technologies, which struggle to simultaneously guarantee structural fidelity, semantic accuracy, and stylistic consistency in sketch-driven generation due to limited conditions or insufficient coupling. It achieves the collaborative use of text prompts and accurate transfer of reference art images while adhering to the hand-drawn sketch, significantly improving the image quality and controllability of sketch-driven art image generation.

[0019] 2. This invention, through a dual-mask structure injection module, distinguishes the multi-scale structure mask generated from the sketch into edge regions and non-edge regions in the feature maps of each layer of the denoising U-Net network, and introduces independent trainable adaptive parameters for feature modulation for each. This solves the problem in existing methods where, when dealing with rough hand-drawn sketches, strengthening constraints restricts artistic expression, while weakening constraints leads to inconsistencies between the result and the sketch structure. It achieves a precise balance between sketch constraints and generation freedom, ensuring that the generated image is faithful to the input sketch outline while retaining rich texture details and artistic expression.

[0020] 3. This invention interacts with the visual features of the reference image through a query transformer to extract a style embedding decoupled from the specific spatial content. Then, the style embedding and text embedding are collaboratively injected into the cross-attention layer of the denoising U-Net network through a content-style adaptive fusion module. This achieves high-quality transfer of the reference art style and solves the technical problem in existing methods where the specific content information carried by the reference image style interferes with the generated image structure and causes the result to deviate from the input sketch. Thus, it can accurately reproduce the target art style while strictly following the layout of the sketch content. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a sketch-driven artistic image generation method proposed in this invention. Figure 2 A schematic diagram of the overall architecture of a network model for generating artistic images; Figure 3 This is a schematic diagram of the structure of a sketch-driven artistic image generation system proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0023] The following English abbreviations are involved: Contrastive Language-Image Pre-training (CLIP) Variational Autoencoder (VAE) Example 1 This embodiment provides a sketch-driven method for generating artistic images, such as... Figure 1 As shown, it includes the following steps: S1. Collect hand-drawn sketches, text prompts, and reference art images provided by the user.

[0024] Hand-drawn sketches are used to provide constraints and definitions for the overall outline, shape structure, spatial layout, and local structural relationships of a target artistic image.

[0025] Text prompts are used to supplement semantic details that are not explicitly expressed in hand-drawn sketches, including object categories, scene descriptions, and other supplementary semantic information.

[0026] Reference art images are used to provide a reference for the artistic style, color distribution, texture features, and brushstroke expression of the target art image.

[0027] S2. Perform image preprocessing on the hand-drawn sketch to obtain a binary sketch image.

[0028] Image preprocessing includes image scaling, edge enhancement, and binarization. The binary sketch image is used to characterize the contour structure and spatial layout information of the target artistic image. The structure of the final generated target artistic image is consistent with the structure of the hand-drawn sketch.

[0029] S3. Input the binary sketch image, text prompt, and reference art image into the pre-trained art image generation network model, and perform an iterative denoising diffusion process in the latent space to obtain the target art image.

[0030] like Figure 2 As shown, the art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from binary sketch images and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of reference art images. The content-style adaptive fusion module fuses text and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising diffusion process based on the multi-scale structure mask and the fused text and style embeddings in the latent space, and gradually recovers the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

[0031] The specific steps for obtaining the target art image by inputting a binary sketch image, text prompt, and reference art image into a pre-trained art image generation network model and performing an iterative denoising diffusion process in the latent space include: Extract the text embedding of the text prompt using a text encoder.

[0032] Extracting text embeddings from text prompts using CLIP text encoder .

[0033] Visual features of the reference art image are extracted using an image encoder.

[0034] Extracting visual features from reference art images using the CLIP image encoder ,in, For global semantic features, The output of the CLIP model Transformer layer is the first Local visual features.

[0035] The binary sketch image is downsampled by a factor of the feature map resolution of different layers in the denoising U-Net network using a dual-mask structure injection module. This generates a multi-scale structure mask corresponding to the feature map size of each network layer, which is then injected into the denoising U-Net network. The multi-scale structure mask is as follows: In the formula, For the first Multi-scale structure mask of layered network layers, For the first The height of the feature maps of the layer network layers, For the first The width of the feature map of a layer in a network; The specific steps for injecting multi-scale structure masks into the denoising U-Net network include: The potential diffusion features of the current network layer of the denoised U-Net network and its corresponding multi-scale structure mask are calculated element-wise to obtain the edge region features and non-edge region features.

[0036] For the denoising U-Net network, the first Layer potential diffusion characteristics ,in, The number of channels; the corresponding multi-scale structure mask. Element-wise computation is performed with the features of this layer to obtain the edge region features. Non-edge region features Edge region features Primarily corresponds to the position of sketch lines, used to strengthen structural constraints; non-edge region features Primarily corresponding to the edge areas of the sketch, it is used to preserve the flexibility of texture generation and artistic expression.

[0037] in, This indicates element-wise multiplication.

[0038] Independent trainable channel scale parameters and bias parameters are set for edge region features and non-edge region features respectively, and adaptive adjustment is performed on edge region features and non-edge region features; Adaptive adjustment of edge region features for: Adaptive adjustment of non-edge region features for: In the formula, , These are the trainable channel scale parameters. , This is the bias parameter.

[0039] The adaptively adjusted edge region features and non-edge region features are fused element-wise and used as the output features of the current network layer. The input is then fed into subsequent network layers.

[0040] The visual features of the reference art image are decoupled by a query transformer to extract the style embedding.

[0041] Since the visual features of the reference art image contain both content and style information, directly using it as input can easily interfere with the target structure provided by the sketch. This embodiment introduces a query transformer (Q-Former) to decouple and extract style embeddings.

[0042] set up A learnable style query vector And query the vector and the visual features of the reference artistic image. Interacting with the Q-Former allows the query vector to aggregate art style-related information from visual features, thereby obtaining a relatively decoupled style embedding representation. The calculation process of Q-Former can be represented as follows: Its internal attention calculation form is as follows: In the formula, , , The projection matrix is ​​learnable. For feature dimensions.

[0043] Through this interactive process, the query vector aggregates style-related information from visual features while minimizing content interference caused by specific spatial layouts.

[0044] The text embedding and style embedding are injected into the denoising U-Net network using a key-value fusion strategy through a content-style adaptive fusion module. The specific steps include: Text embedding and style embedding are mapped separately using independent learnable linear projection matrices to obtain their respective key and value representations; Given style embedding and text embedding Flatten the current network layer features of U-Net as .

[0045] Style-embedded key representation Sum value representation for: Key representation of text embedding Sum value representation for: In the formula, A learnable linear projection matrix for the key representation of style embedding. A learnable linear projection matrix representing the value of style embedding. A learnable linear projection matrix for the key representation of text embeddings. A learnable linear projection matrix for the key representation of text embeddings.

[0046] Style embedding is used as a supplement to text embedding, and concatenation is performed at the token dimension to obtain the merged key representation. Sum value representation : In the formula, This is for splicing operations.

[0047] In each cross-attention layer of the denoising U-Net network, the feature map of the current layer is flattened and used as a query. Cross-attention calculation is then performed with the fused key and value representations, thereby injecting text content and style information into the diffusion process of the denoising U-Net network.

[0048] The expression for cross-attention calculation is: The denoising U-Net network predicts noise and gradually recovers the target latent variables corresponding to the target artistic image during the iterative denoising process based on random noise, multi-scale structural masks, text embeddings, and style embeddings. Specific steps include: A random noise is generated by sampling in the potential space as the starting point for the diffusion process; The noise-added latent features at the current time step are input into the denoising U-Net network, and the noise components contained in the noise-added latent features at the current time step are predicted by combining multi-scale structure mask, text embedding and style embedding. The denoising latent features at the current time step are updated based on the noise components, and the updated denoising latent features are used as the input for the next round of denoising. When the iterative denoising loop reaches time step 0, the diffusion process terminates, and the target latent variable corresponding to the target artistic image is obtained.

[0049] The target latent variables are decoded using a VAE decoder to obtain the target artistic image. The target artistic image maintains consistency with the input sketch in overall structure, with the text prompts in semantic content, and with the reference artistic image in artistic style.

[0050] In the process of training the art image generation network model, the original image is first obtained from the public text to image generation dataset. The sketch generation model is used to extract the structural contour information from the original image to generate the corresponding sketch image. The reference style image is obtained from the art style dataset. The style transfer model is used to convert the original image into an image with the specified art style. The sketch structure image, the corresponding text description and the reference style image are paired to form training samples.

[0051] The training images are encoded into the latent space using a VAE encoder to obtain latent features. and at random time steps Downward latent features Add Gaussian noise To obtain the latent features of noise addition .

[0052] Potential features of noise addition Input the denoising U-Net network and simultaneously mask the sketch structure. The input double-mask structure injection module embeds text semantics. With style embedding The input adaptive content style fusion module uses a denoising U-Net network to predict noise components. .

[0053] The art image generation network model is trained using the diffusion reconstruction loss function. During training, the main parameters of the model are frozen, and only the trainable parameters in the double-mask structure injection module, Q-Former, and content-style adaptive fusion module are trained. Among them, the content-style adaptive fusion module preferably uses the LoRA method for lightweight fine-tuning.

[0054] The diffusion reconstruction loss function is: In the formula, For the diffusion reconstruction loss function, For mathematical expectation, As a potential feature, It is Gaussian noise. For reference to the visual characteristics of artistic images, For multi-scale structure masks, The noise components predicted by the denoising U-Net network. For time steps The potential features of noise addition For text prompts, is the time step; where the latent features are obtained by encoding the training images into the latent space using a VAE encoder.

[0055] To improve the consistency between the generated image and multiple conditions, a multi-condition classification-free guidance strategy is adopted during training. This strategy controls the alignment between the generated content and text semantics, style features, and sketch structure by adjusting the weights of different conditions. Noise estimation after guidance is also performed. Represented as: In the formula, , , The guiding weights are for text, style, and sketch, respectively.

[0056] Example 2 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the sketch-driven artistic image generation method provided in Embodiment 1.

[0057] The rest is the same as in Example 1.

[0058] Example 3 This embodiment provides a sketch-driven artistic image generation system, such as... Figure 3 As shown, it includes: The data acquisition module is used to collect hand-drawn sketches, text prompts, and reference art images provided by the user; The sketch processing module is used to preprocess hand-drawn sketches to obtain binary sketch images. The art image generation module is used to input binary sketch images, text prompts, and reference art images into a pre-trained art image generation network model, and perform an iterative denoising diffusion process in the latent space to obtain the target art image; The art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from binary sketch images and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of reference art images. The content-style adaptive fusion module fuses text and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising process in the latent space based on the multi-scale structure mask and the fused text and style embeddings, gradually recovering the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

[0059] The rest is the same as in Example 1.

[0060] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A sketch-driven artistic image generation method, characterized in that, Includes the following steps: Collect hand-drawn sketches, text prompts, and reference art images provided by users; The hand-drawn sketch is preprocessed to obtain a binary sketch image; The binary sketch image, text prompt, and reference art image are input into a pre-trained art image generation network model, and an iterative denoising diffusion process is performed in the latent space to obtain the target art image. The art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from the binary sketch image and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of the reference art image. The content-style adaptive fusion module fuses the text embeddings and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising diffusion process based on the multi-scale structure mask and the fused text and style embeddings in the latent space, and gradually recovers the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

2. The sketch-driven artistic image generation method according to claim 1, characterized in that, The text prompts are used to supplement semantic details that are not explicitly expressed in the hand-drawn sketches, including object categories, scene descriptions, and other supplementary semantic information. The reference art images are used to provide references for the artistic style, color distribution, texture features, and brushstroke expression of the target art image.

3. The sketch-driven artistic image generation method according to claim 1, characterized in that, The image preprocessing includes image scaling, edge enhancement, and binarization. The binary sketch image is used to characterize the contour structure and spatial layout information of the target art image, and the structure of the target art image is consistent with the structure of the hand-drawn sketch.

4. The sketch-driven artistic image generation method according to claim 1, characterized in that, The specific steps for inputting the binary sketch image, text prompt, and reference artistic image into a pre-trained artistic image generation network model, and performing an iterative denoising diffusion process in the latent space to obtain the target artistic image include: Extract the text embedding of the text prompt using a text encoder; Visual features of the reference art image are extracted using an image encoder; The binary sketch image is downsampled by a factor of the feature map resolution of different network layers in the denoising U-Net network through the dual mask structure injection module, generating a multi-scale structure mask corresponding to the feature map size of each network layer, and then injected into the denoising U-Net network. The visual features of the reference art image are decoupled by a query transformer to extract the style embedding. The text embedding and style embedding are injected into the denoising U-Net network using a key-value fusion strategy through the content style adaptive fusion module; The denoising U-Net network predicts noise during the diffusion process of iterative denoising based on random noise, multi-scale structure mask, text embedding, and style embedding, and gradually recovers the target latent variables corresponding to the target artistic image. The target latent variables are decoded using a VAE decoder to obtain the target artistic image.

5. The sketch-driven artistic image generation method according to claim 4, characterized in that, The multi-scale structure mask is: In the formula, For the first Multi-scale structure mask of layered network layers, For the first The height of the feature maps of the layer network layers, For the first The width of the feature map of a layer in a network; The specific steps for injecting the multi-scale structure mask into the denoising U-Net network include: The potential diffusion features of the current network layer of the denoised U-Net network and its corresponding multi-scale structure mask are calculated element-wise to obtain edge region features and non-edge region features. Independent trainable channel scale parameters and bias parameters are set for the edge region features and non-edge region features respectively, and the edge region features and non-edge region features are adaptively adjusted. After adaptively adjusting the edge region features and non-edge region features, they are fused element-wise and then used as the output features of the current network layer to be input into subsequent network layers.

6. The sketch-driven artistic image generation method according to claim 4, characterized in that, The specific steps of injecting the text embedding and style embedding into the denoising U-Net network using a key-value fusion strategy through the content-style adaptive fusion module include: The text embedding and style embedding are mapped using independent learnable linear projection matrices to obtain their respective key and value representations; Style embedding is used as a supplement to text embedding, and the two are concatenated at the token dimension to obtain the fused key and value representations. In each cross-attention layer of the denoising U-Net network, the feature map of the current layer is flattened and used as a query. Cross-attention calculation is then performed with the fused key and value representations, thereby injecting text content and style information into the diffusion process of the denoising U-Net network.

7. The sketch-driven artistic image generation method according to claim 4, characterized in that, The specific steps of using the denoising U-Net network to predict noise and gradually recover the target latent variables corresponding to the target artistic image during the iterative denoising process based on random noise, multi-scale structural masks, text embeddings, and style embeddings include: A random noise is generated by sampling in the potential space as the starting point for the diffusion process; The noise-added latent features at the current time step are input into the denoising U-Net network, and the noise components contained in the noise-added latent features at the current time step are predicted by combining multi-scale structure mask, text embedding and style embedding. The noise latent features at the current time step are updated based on the noise components, and the updated noise latent features are used as the input for the next round of denoising. When the iterative denoising loop reaches time step 0, the diffusion process terminates, and the target latent variable corresponding to the target artistic image is obtained.

8. The sketch-driven artistic image generation method according to claim 1, characterized in that, The art image generation network model is trained using a diffusion reconstruction loss function, which is: In the formula, For the diffusion reconstruction loss function, For mathematical expectation, As a potential feature, It is Gaussian noise. For reference to the visual characteristics of artistic images, For multi-scale structure masks, The noise components predicted by the denoising U-Net network. For time steps The potential features of noise addition For text prompts, For time steps; The latent features are obtained by encoding the training images into the latent space using a VAE encoder.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the sketch-driven artistic image generation method as described in any one of claims 1 to 8.

10. A sketch-driven artistic image generation system, characterized in that, include: The data acquisition module is used to collect hand-drawn sketches, text prompts, and reference art images provided by the user; The sketch processing module is used to perform image preprocessing on the hand-drawn sketch to obtain a binary sketch image; The art image generation module is used to input the binary sketch image, text prompt, and reference art image into a pre-trained art image generation network model to obtain the target art image; The art image generation network model includes a VAE encoder, a text encoder, an image encoder, a dual-mask structure injection module, a query transformer, a content-style adaptive fusion module, a denoising U-Net network, and a VAE decoder. The VAE encoder encodes training images into the latent space during model training. The text encoder extracts text embeddings from text prompts. The image encoder extracts visual features from reference art images. The dual-mask structure injection module generates multi-scale structure masks from the binary sketch image and injects them into the denoising U-Net network. The query transformer extracts style embeddings from the visual features of the reference art image. The content-style adaptive fusion module fuses the text embeddings and style embeddings and injects them into the denoising U-Net network. The denoising U-Net network predicts noise during the iterative denoising diffusion process based on the multi-scale structure mask and the fused text and style embeddings in the latent space, and gradually recovers the target latent variables corresponding to the target art image. The VAE decoder decodes the target latent variables from the latent space and reconstructs the target art image.

Images