A stroke consistency sketch extraction method and device based on large model interaction
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAQIAO UNIVERSITY
- Filing Date
- 2026-07-08
- Publication Date
- 2026-08-04
AI Technical Summary
但该先进思路尚未应用于素描提取领域，业内亟需一种兼顾内容完整性、艺术笔触质感，且支持自然语言交互微调的高效素描生成方案
本发明具有如下有益效果：首先，本发明实现了笔触高度一致性还原，借助大语言模型文本描述辅助引导，模型能够精准感知并复刻参考图像笔触的粗细、肌理材质等物理特征，使生成画面在笔触质感、绘画肌理上与参考风格高度契合，避免笔触风格错乱脱节的问题。其次，实现图像语义与结构双重保真，通过引入前景掩码与结构图信息注入策略，有效克服传统风格迁移算法易出现的主体轮廓变形、关键内容丢失、空间结构崩坏等缺陷，完整保留原图主体语义信息与整体构图结构，提升生成结果的合理性与真实度。最后，本发明具备零样本、免训练的突出特性，无需针对各类特定画风单独训练 LoRA 模型或进行模型微调，无需额外专项数据集适配，可做到即插即用，对未知小众画风、自定义艺术风格均具备强适应性与泛化能力，大幅降低应用部署门槛与使用成本。
Smart Images
Figure CN122510397A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and generative artificial intelligence, and in particular provides a method and apparatus for extracting stroke consistency sketches based on large model interaction. Background Technology
[0002] Sketching, with its core artistic features of rhythmic lines and textured brushstrokes, demands a high level of detail in both the quality of the lines and the integrity of the content's structure. With the development of artificial intelligence technology, algorithm-generated sketching solutions are widely used. Traditional edge extraction techniques can only output basic outlines, resulting in monotonous, rigid lines and a lack of artistic texture, making it difficult to meet the needs of generating artistic sketches.
[0003] Current style transfer methods based on diffusion models can achieve preliminary stylized rendering of sketches, but they still have significant technical shortcomings. First, existing methods do not explicitly model brushstrokes, relying solely on pixel statistical features for style transfer, failing to capture personalized artistic brushstrokes and resulting in highly homogenized works with insufficient artistic expression. Second, the models suffer from the problem of content and style features becoming entangled; forcibly transferring styles can easily lead to distortion of the original image's main structure and content collapse, and cluttered background textures can easily obscure the main content, destroying the integrity and depth of the image. Third, existing solutions have weak interactivity, unable to achieve fine-tuning of style and brushstrokes through natural language, making it difficult to adapt to personalized creative needs.
[0004] Current cutting-edge proxy-perspective image generation paradigms rely on large models to first understand the creative intent behind an image and then guide the generation process, effectively compensating for the shortcomings of traditional generation models. However, this advanced approach has not yet been applied to the field of sketch extraction. The industry urgently needs a highly efficient sketch generation solution that balances content integrity, artistic brushstroke quality, and supports fine-tuning through natural language interaction. Summary of the Invention
[0005] In view of the above problems, the present invention relates to a method for extracting brushstroke consistency sketches based on large model interaction, comprising the following steps:
[0006] S1 acquires the input image and performs multimodal preprocessing: It acquires the content image and a reference sketch image, extracts the structure map of the content image using an edge detection algorithm, generates a brushstroke style description for the reference sketch image using a multimodal large model, and identifies foreground objects in the content image to obtain style cues. and object prompts ; S2 Constructing the latent space inversion: Using a pre-trained diffusion model, combined with the structure graph as a condition, denoising diffusion inversion is performed on the content image and the reference sketch image to obtain their respective latent noise representations; S3 constructs a cross-modal brushstroke attention mechanism: In the U-Net network layer of the diffusion model, the key features K and value features V of the reference sketch are injected into the self-attention calculation of content image generation, and style cues are fused. The text embedding features are used to style-align the key and value features of the reference sketch image, generating intermediate features that fuse brushstroke attributes. and style tips Input the text encoder to obtain style text embedding features; S4 Application Prospect Focusing Guidance Module: Performs spatial clustering on self-attention maps, combined with object prompts. Cross-attention maps generate foreground masks, restricting brushstroke style features to be injected only in the foreground region and suppressing background textures; S5 Semantic Preservation and Stroke Refinement: Adjusting Query Vectors by Injecting Latent Features into the Structure Graph The semantic loss function is used to optimize the generated results, and finally the classifier is used to freely guide the fusion of text semantics and visual brushstroke features to obtain the final predicted noise. S6 Output and Evaluation: Decode the final latent representation to obtain a sketch image and perform a quality evaluation.
[0007] Preferably, the step S1 of extracting the structure map of the content image using the edge detection algorithm specifically includes the following steps: normalizing the size of the content image to match the input size of the pre-trained diffusion model; converting the size-normalized content image into a grayscale image and denoising the grayscale image to obtain a preprocessed image; calculating the edge gradient of the preprocessed image to obtain the gradient magnitude and gradient direction of each pixel; performing non-maximum suppression based on the gradient magnitude and gradient direction to obtain candidate edge pixels; setting a threshold for the edge gradient magnitude, and performing double-threshold edge connection on the candidate edge pixels based on the high and low thresholds of the edge gradient magnitude threshold to determine strong edge pixels and weak edge pixels connected to the strong edge pixels, generating an initial edge map; performing morphological post-processing on the initial edge map to obtain a continuous content structure contour; and performing size matching and pixel value normalization on the content structure contour to obtain the structure map of the content image.
[0008] Preferably, the specific process of generating brushstroke style descriptions using a multimodal large model in S1 is as follows: A reference sketch image is input into a visual language model. A preset prompt template is used to query the image's line type, hatching method, and overall artistic style. A textual description is generated of the reference sketch image's line type, line thickness, hatching method, brushstroke density, light and shadow levels, and overall artistic style. The textual description is then structured using a field-based processing module to output a structured style prompt. .
[0009] Preferably, the visual language model is used for image understanding and text description generation of the reference sketch image, and the visual language model includes an image feature extraction module, an image-text semantic alignment module, a text generation module, and a field organization module; the image feature extraction module extracts line shape features, hatching texture features, light and dark level features, and overall composition features from the reference sketch image; the image-text semantic alignment module maps the line shape features, hatching texture features, light and dark level features, and overall composition features to the text semantic space.
[0010] Preferably, obtaining the respective latent noise representations in S2 includes the following steps: First, the content image and the reference sketch image are respectively input into the variational autoencoder of the pre-trained diffusion model to obtain the initial latent representations of the content image and the reference sketch image; Second, the structure map of the content image is encoded to obtain the structural condition features, and the structural condition features are input into the denoising network of the diffusion model; Then, according to the preset diffusion time step, the initial latent representations are gradually back-introduced from the low-noise latent state to the high-noise latent state using the diffusion inversion method to obtain the latent noise representations corresponding to the content image and the reference sketch image, respectively. The latent noise representations are used for feature extraction and generation initialization in the subsequent cross-modal stroke attention mechanism.
[0011] Preferably, the calculation formula for the cross-modal stroke attention mechanism in S3 is as follows: Set content image The query features are The key features of the content image are The value features of the content image are The key features of the reference sketch are The value characteristics of the reference sketch are ; Mixed bond features Sum value characteristics The calculation formula is:
[0012] , in, and Key features of the reference sketch image Sum value characteristics Injected into content image key features Sum value characteristics The injection mixing ratio coefficient in the formula has a range of values of 100%. ; This indicates an adaptive instance normalization operation, used to normalize style hints. The text embedding statistical features are aligned to the reference image features.
[0013] Preferably, the specific steps of applying the foreground focusing guidance module in S4 include: Self-attention feature map K-Means clustering was performed to obtain Cluster ,in , To obtain the total number of clusters by clustering; Calculate each cluster With object prompts Corresponding cross-attention map correlation score :
[0014] in For pixel coordinates, To prevent tiny constants with a denominator of zero, Multiply pixel by pixel; set a threshold Only retain those that meet the requirements. Clusters as foreground region masks In subsequent attention calculations, the injection weights of reference features in non-foreground regions are reset to zero, thereby suppressing background texture.
[0015] Preferably, in step S5, the query vector is adjusted by injecting latent features from the structure graph. The specific steps for optimizing the generated results using a semantic loss function are as follows:
[0016] in, These are the query features extracted from the structure graph. This is a structural control factor used to adjust the strength of the contour constraints; The semantic loss function Defined as the cosine distance between the generated image and the content image in the CLIP feature space:
[0017] in This indicates the CLIP image encoder. This indicates the generation of a sketch image. This represents the input image, i.e., the original image that needs to be converted into a sketch. This represents the cosine similarity function, used to measure the degree of similarity between the feature vectors of two CLIP images.
[0018] Preferably, in step S5, the final predicted noise obtained by freely guiding the fusion of text semantics and visual brushstroke features through a classifier is:
[0019] in, To ultimately predict noise, For noise prediction networks, For the potential variables at the current time step, The condition is empty. This serves as a text-guided scale. This serves as a guide scale for the image. For reference sketch image.
[0020] Preferably, the multimodal large model in S1 adopts GPT-4o, and the pre-trained diffusion model in S2 adopts the StableDiffusion model.
[0021] Preferably, the evaluation metrics in S6 include: ArtFID: used to evaluate the consistency between the generated sketch and the distribution of the reference art style; LPIPS: used to evaluate the perceptual similarity between the generated sketch and the content image; FID: used to evaluate the overall realism and distribution distance of the generated image.
[0022] A brushstroke consistency sketch extraction device based on large model interaction includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This invention offers the following advantages: First, it achieves highly consistent brushstroke reproduction. Assisted by textual descriptions from a large language model, the model accurately perceives and replicates the physical characteristics of the brushstrokes in the reference image, such as thickness and texture. This ensures the generated image closely matches the reference style in terms of brushstroke texture and painting quality, avoiding stylistic inconsistencies. Second, it achieves dual fidelity in image semantics and structure. By introducing foreground masking and structural graph information injection strategies, it effectively overcomes the shortcomings of traditional style transfer algorithms, such as subject contour deformation, loss of key content, and spatial structure collapse. It fully preserves the semantic information of the original image and the overall compositional structure, improving the rationality and realism of the generated results. Finally, this invention features zero-sample, training-free operation. It eliminates the need for separate training of LoRA models or model fine-tuning for various specific art styles, and requires no additional specialized dataset adaptation. It is plug-and-play, exhibiting strong adaptability and generalization capabilities for unknown niche art styles and custom art styles, significantly reducing the application deployment threshold and usage costs. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating the overall process architecture of the method of this invention.
[0025] Figure 2 This is a schematic diagram of the cross-modal stroke attention mechanism of the present invention.
[0026] Figure 3 This is a schematic diagram illustrating the principle of the cross-modal brushstroke content image structure extraction process of this invention.
[0027] Figure 4 This is a schematic diagram of the cross-modal brushstroke visual latent space inversion process of the present invention. Figure 5 This is a schematic diagram of the cross-modal brushstroke visual language model of the present invention.
[0028] Figure 6 This is a schematic diagram illustrating the principle of cross-modal stroke attention injection in this invention.
[0029] Figure 7 This is a schematic diagram of the cross-modal stroke quality assessment model of the present invention. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.
[0031] Example The following are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the following embodiments. All technical solutions that fall within the scope of the present invention are within the scope of protection of the present invention.
[0032] Reference manual attached Figure 1-7This invention provides a method for extracting sketches with consistent brushstrokes based on large model interaction, comprising the following steps: S1 acquires the input image and performs multimodal preprocessing: It acquires the content image and a reference sketch image, extracts the structure map of the content image using an edge detection algorithm, generates a brushstroke style description for the reference sketch image using a multimodal large model, and identifies foreground objects in the content image to obtain style cues. and object prompts The specific process of generating brushstroke style descriptions using a multimodal large model is as follows: A reference sketch image is input into the visual language model. A preset prompt template is used to query the image for line type, hatching method, and overall artistic style, outputting structured style prompts. .
[0033] S2 Constructing the latent space inversion: Using a pre-trained diffusion model, combined with the structure graph as a condition, denoising diffusion inversion is performed on the content image and the reference sketch image to obtain their respective latent noise representations; S3 constructs a cross-modal stroke attention mechanism: see attached instruction manual. Figure 2 In the U-Net network layer of the diffusion model, the key features K and value features V of the reference sketch are injected into the self-attention calculation for content image generation, and style cues are fused. The text embedding features are used to generate intermediate features that fuse brushstroke attributes; In the self-attention layer of U-Net:
[0034] This invention modifies and Not just using the key-value pairs of the content image Instead, it incorporates features from the reference image. To enhance stroke consistency, this invention introduces an adaptive instance normalization mechanism for text guidance: Features of the reference image Before being injected, the style description is generated based on the large model. Feature alignment was performed to ensure that stroke features were injected. Therefore, the calculation formula for the cross-modal stroke attention mechanism is as follows: Set content image The query features are The key features of the content image are The value features of the content image are The key features of the reference sketch are The value characteristics of the reference sketch are ; Mixed bond features Sum value characteristics The calculation formula is:
[0035] , in, and Key features of the reference sketch image Sum value characteristics Injected into content image key features Sum value characteristics The injection mixing ratio coefficient in the formula has a range of values of 100%. ; This indicates an adaptive instance normalization operation, used to normalize style hints. The text embedding statistical features are aligned to the reference image features.
[0036] S4 Application Prospect Focusing Guidance Module: Performs spatial clustering on self-attention maps, combined with object prompts. Cross-attention maps generate foreground masks, restricting brushstroke style features to be injected only in the foreground region and suppressing background textures; The specific steps of the application prospect-focused guidance module include: Self-attention feature map K-Means clustering was performed to obtain Cluster ,in , To obtain the total number of clusters by clustering; Calculate each cluster With object prompts Corresponding cross-attention map correlation score :
[0037] in For pixel coordinates, To prevent the use of tiny constants with a denominator of zero, a threshold is set. Only retain those that meet the requirements. Clusters as foreground region masks In subsequent attention calculations, the injection weights of reference features in non-foreground regions are reset to zero, thereby suppressing background texture.
[0038] S5 Semantic Preservation and Stroke Refinement: Adjusting Query Vectors by Injecting Latent Features into the Structure Graph The semantic loss function is used to optimize the generated results, and finally, a classifier is used to freely guide the fusion of textual semantics and visual brushstroke features; the query vector is adjusted by injecting latent features from the structure graph. The specific steps for optimizing the generated results using a semantic loss function are as follows:
[0039] in, These are the query features extracted from the structure graph. This is a structural control factor used to adjust the strength of the contour constraints; The semantic loss function Defined as the cosine distance between the generated image and the content image in the CLIP feature space:
[0040] in This indicates the CLIP image encoder. This indicates the CLIP image encoder. This indicates the generation of a sketch image. This represents the input image, i.e., the original image that needs to be converted into a sketch. This represents the cosine similarity function, used to measure the similarity between the feature vectors of two CLIP images.
[0041] The final predicted noise obtained by freely guiding the fusion of text semantics and visual brushstroke features through a classifier in S5 is:
[0042] in, To ultimately predict noise, For noise prediction networks, For the potential variables at the current time step, The condition is empty. This serves as a text-guided scale. This serves as a guide scale for the image. For reference sketch image.
[0043] S6 Output and Evaluation: Decode the final latent representation to obtain a sketch image and perform a quality evaluation.
[0044] Specific evaluation metrics include: ArtFID: used to evaluate the consistency of the generated sketch's style distribution with the reference art style distribution; LPIPS: used to evaluate the perceptual similarity between the generated sketch and the content image; FID: used to evaluate the overall realism and distribution distance of the generated image. A set of generated sketch images, a set of reference sketch images, and a set of content images are obtained, and their sizes are unified and pixel values are normalized. The generated sketch image set and the reference sketch image set are input into a pre-trained art style feature extraction network to extract art style features. The ArtFID index is calculated based on the mean and covariance of the two sets of art style features to evaluate the consistency of the generated sketch image and the reference sketch image in terms of art style distribution. A smaller ArtFID value indicates a closer similarity in brushstroke style between the generated sketch image and the reference sketch image. The generated sketch image and the corresponding content image are then input into a pre-trained perceptual feature network. The network extracts multilayer perceptual features and calculates the distance between the two in the perceptual feature space to obtain the LPIPS index, which is used to evaluate the degree to which the generated sketch image preserves the structural and semantic information of the content image. The smaller the LPIPS value, the higher the perceptual similarity between the generated sketch image and the content image. The generated sketch image set and the real sketch image set are input into the pre-trained image feature extraction network to extract image distribution features respectively. The FID index is calculated based on the mean and covariance of the two sets of image distribution features to evaluate the distance between the overall distribution of the generated sketch image and the distribution of the real sketch image. The smaller the FID value, the higher the overall realism of the generated sketch image.
[0045] Reference manual attached Figure 1 As shown, this invention provides a method for extracting stroke consistency sketches based on large model interaction. The overall process includes: first, acquiring a content image and a reference sketch image; extracting a structure map from the content image; generating a stroke style description from the reference sketch image; and identifying foreground objects in the content image; second, performing latent space inversion on the content image and the reference sketch image based on a pre-trained diffusion model; then, constructing a cross-modal stroke attention mechanism in the U-Net network layer of the diffusion model to inject stroke features from the reference sketch image into the content image generation process; subsequently, using a foreground focusing guidance module to restrict the injection of stroke features within the foreground region; and finally, obtaining the final sketch image through semantic preservation, stroke refinement, and quality evaluation.
[0046] Reference manual attached Figure 2As shown, the cross-modal brushstroke attention mechanism includes a content image feature branch, a reference sketch feature branch, and a style cue text branch. The content image feature branch extracts query features, key features, and value features from the content image; the reference sketch feature branch extracts key features and value features from the reference sketch image; and the style cue text branch provides semantic constraints on brushstroke style. By mixing the key and value features of the reference sketch image with the corresponding features of the content image, the generation process incorporates the brushstroke style of the reference sketch image while maintaining the structure of the content image.
[0047] Reference manual attached Figure 3 As shown, the content image structure map extraction process includes: inputting the content image into the content image input module, performing size normalization and grayscale denoising processing through the preprocessing module; then performing edge detection on the preprocessed image through the edge detection extraction module, and sequentially completing gradient magnitude and direction calculation, non-maximum suppression, and double threshold edge connection in the structure map generation module; finally, performing morphological post-processing on the generated initial edge map to output the content image structure map.
[0048] Reference manual attached Figure 4 As shown, the latent space inversion process includes a content image input branch, a reference sketch image input branch, and a structural condition input branch. The content image and reference sketch image are encoded by a variational autoencoder to obtain initial latent representations; the structural image is encoded by a structural condition encoding module to obtain structural condition features; subsequently, the initial latent representations and structural condition features are input into a diffusion inversion module, which progressively back-engineers the process to a high-noise latent state according to a preset diffusion time step, obtaining the latent noise representations corresponding to the content image and the reference sketch image, respectively.
[0049] Reference manual attached Figure 5 As shown, the visual language model is used to generate style prompts. Specifically, after inputting a reference sketch image into the visual language model, the image feature extraction module first extracts line shape features, hatching texture features, light and dark level features, and overall composition features; then, the image-text semantic alignment module maps the visual features to the text semantic space; subsequently, the text generation module generates a brushstroke style description based on a preset prompt template; finally, the field processing module performs structured processing on the brushstroke style description and outputs the style prompt Pstyle.
[0050] Reference manual attached Figure 6As shown, the cross-modal stroke attention injection process includes input of content image features, style cue input, and reference sketch features. The content image provides query features, key features, and value features; the reference sketch image provides reference key features and reference value features; the style cue is text-encoded to obtain style text embedding features, and the reference key features and reference value features are style-aligned through an adaptive instance normalization module. Then, according to the injection ratio coefficient, the style-aligned reference key features and reference value features are injected into the corresponding key features and value features of the content image, respectively, to obtain mixed key features and mixed value features. Attention is then calculated based on the query features, mixed key features, and mixed value features of the content image to generate intermediate features that fuse stroke attributes.
[0051] Reference manual attached Figure 7 As shown, the quality assessment process includes image preprocessing, ArtFID evaluation, LPIPS evaluation, and FID evaluation. First, the generated sketch image set, reference sketch image set, content image set, and real sketch image set are obtained and their dimensions and pixel values are normalized. Then, an ArtFID index is calculated using a pre-trained art style feature extraction network to evaluate the consistency of the generated sketch images and reference sketch images in terms of art style distribution. An LPIPS index is calculated using a pre-trained perceptual feature extraction network to evaluate the degree to which the generated sketch images preserve the structural and semantic information of the content images. Finally, an FID index is calculated using a pre-trained image feature extraction network to evaluate the distance between the overall distribution of the generated sketch images and the distribution of the real sketch images.
[0052] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A method for extracting sketches with consistent brushstrokes based on large-scale model interaction, characterized in that, Includes the following steps: S1 acquires the input image and performs multimodal preprocessing: It acquires the content image and a reference sketch image, extracts the structure map of the content image using an edge detection algorithm, generates a brushstroke style description for the reference sketch image using a multimodal large model, and identifies foreground objects in the content image to obtain style cues. and object prompts ; S2 Constructing the latent space inversion: Using a pre-trained diffusion model, combined with the structure graph as a condition, denoising diffusion inversion is performed on the content image and the reference sketch image to obtain their respective latent noise representations; S3 constructs a cross-modal brushstroke attention mechanism: In the U-Net network layer of the diffusion model, the key features K and value features V of the reference sketch are injected into the self-attention calculation of content image generation, and style cues are fused. The text embedding features are used to style-align the key and value features of the reference sketch image, generating intermediate features that fuse brushstroke attributes. and style tips Input the text encoder to obtain style text embedding features; S4 Application Prospect Focusing Guidance Module: Performs spatial clustering on self-attention maps, combined with object prompts. Cross-attention maps generate foreground masks, restricting brushstroke style features to be injected only in the foreground region and suppressing background textures; S5 Semantic Preservation and Stroke Refinement: Adjusting Query Vectors by Injecting Latent Features into the Structure Graph The semantic loss function is used to optimize the generated results, and finally the classifier is used to freely guide the fusion of text semantics and visual brushstroke features to obtain the final predicted noise. S6 Output and Evaluation: Decode the final latent representation to obtain a sketch image and perform a quality evaluation.
2. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The step S1, which uses an edge detection algorithm to extract the structure map of the content image, specifically includes the following steps: normalizing the size of the content image to match the input size of the pre-trained diffusion model; converting the size-normalized content image into a grayscale image and denoising the grayscale image to obtain a preprocessed image. Edge gradient calculation is performed on the preprocessed image to obtain the gradient magnitude and gradient direction of each pixel. Non-maximum suppression is performed based on the gradient magnitude and gradient direction to obtain candidate edge pixels. An edge gradient magnitude threshold is preset, and double-threshold edge connection is performed on the candidate edge pixels based on the high threshold and the low threshold to determine strong edge pixels and weak edge pixels connected to the strong edge pixels, generating an initial edge map. Morphological post-processing is performed on the initial edge map to obtain a continuous content structure contour. Size matching and pixel value normalization are performed on the content structure contour to obtain the structure map of the content image.
3. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The multimodal large model in S1 adopts GPT-4o. The specific process of generating brushstroke style descriptions using the multimodal large model in S1 is as follows: the reference sketch image is input into the visual language model, and the image's line type, hatching method, and overall artistic style are queried through a preset prompt template. A text description of the reference sketch image's line type, line thickness, hatching method, brushstroke density, light and shadow levels, and overall artistic style is generated. The text description is then structured through a field processing module to output a structured style prompt. ; The visual language model is used for image understanding and text description generation of reference sketch images. The visual language model includes an image feature extraction module, an image-text semantic alignment module, a text generation module, and a field organization module. The image feature extraction module extracts line shape features, hatching texture features, light and dark level features, and overall composition features from the reference sketch image. The image-text semantic alignment module maps the line shape features, hatching texture features, light and dark level features, and overall composition features to the text semantic space.
4. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The pre-trained diffusion model in S2 adopts the StableDiffusion model. The acquisition of the respective latent noise representations in S2 includes the following steps: First, the content image and the reference sketch image are respectively input into the variational autoencoder of the pre-trained diffusion model to obtain the initial latent representations of the content image and the reference sketch image; Second, the structure map of the content image is encoded to obtain the structural condition features, and the structural condition features are input into the denoising network of the diffusion model; Then, according to the preset diffusion time step, the initial latent representation is gradually back-introduced from the low-noise latent state to the high-noise latent state using the diffusion inversion method to obtain the latent noise representations corresponding to the content image and the reference sketch image, respectively. The latent noise representations are used for feature extraction and generation initialization in the subsequent cross-modal stroke attention mechanism.
5. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The calculation formula for the cross-modal stroke attention mechanism in S3 is as follows: Set content image The query features are The key features of the content image are The value features of the content image are The key features of the reference sketch are The value characteristics of the reference sketch are ; Mixed bond features Sum value characteristics The calculation formula is: ， in, and Key features of the reference sketch image Sum value characteristics Injected into content image key features Sum value characteristics The injection mixing ratio coefficient in the formula has a range of values. ; This indicates an adaptive instance normalization operation, used to normalize style hints. The text embedding statistical features are aligned to the reference image features.
6. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The specific steps of applying the foreground focusing guidance module in S4 include: Self-attention feature map K-Means clustering was performed to obtain Cluster ,in , To obtain the total number of clusters by clustering; Calculate each cluster With object prompts Corresponding cross-attention map correlation score : in For pixel coordinates, To prevent tiny constants with a denominator of zero, Multiply pixel by pixel; set a threshold Only retain those that meet the requirements. Clusters as foreground region masks In subsequent attention calculations, the injection weights of reference features in non-foreground regions are reset to zero, thereby suppressing background texture.
7. The method for extracting stroke consistency sketches based on large model interaction according to claim 5, characterized in that, In step S5, the query vector is adjusted by injecting latent features from the structure graph. The specific steps for optimizing the generated results using a semantic loss function are as follows: in, These are the query features extracted from the structure graph. This is a structural control factor used to adjust the strength of the contour constraints; The semantic loss function Defined as the cosine distance between the generated image and the content image in the CLIP feature space: in This indicates the CLIP image encoder. This indicates the generation of a sketch image. This represents the input image, i.e., the original image that needs to be converted into a sketch. This represents the cosine similarity function, used to measure the similarity between the feature vectors of two CLIP images.
8. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The final predicted noise obtained by freely guiding the fusion of text semantics and visual brushstroke features through a classifier in S5 is: in, To ultimately predict noise, For noise prediction networks, For the potential variables at the current time step, The condition is empty. This serves as a text-guided scale. This serves as a guide scale for the image. For reference sketch image.
9. The method for extracting stroke consistency sketches based on large model interaction according to claim 1, characterized in that, The evaluation metrics in S6 include: ArtFID: used to evaluate the consistency between the generated sketch and the distribution of the reference art style; LPIPS: used to evaluate the perceptual similarity between the generated sketch and the content image; FID: used to evaluate the overall realism and distribution distance of the generated image.
10. A sketch extraction device based on large model interaction, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1-9.