Vector-based hand-drawn sketch synthesis method, computer device, and program product
Through a vector-based hand-drawn sketch synthesis method, the diffusion model and text-described edge map features are used to optimize the hand-drawn sketch, which solves the problems of artifacts and inaccurate structural semantic description when generating hand-drawn sketches in the existing technology, and achieves higher quality sketch generation.
Patent Information
- Application Number
- CN202411316919.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2044-09-20
AI Technical Summary
Existing pixel- and vector-based sketch synthesis methods suffer from artifacts, discordant patterns, and inability to accurately describe the content image structure and semantics when generating hand-drawn sketches.
Through a vector-based hand-drawn sketch synthesis method, the diffusion model is combined with text descriptions and edge map features to guide the generation of natural images, and the hand-drawn sketches are optimized through difference noise to improve structural and semantic similarity.
The generated hand-drawn sketch can more accurately describe the detailed information of the content image and improve the similarity of the content structure and semantic structure between the hand-drawn sketch and the content image.
Smart Images
Figure CN119477713B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a vector-based hand-drawn sketch synthesis method, computer device, and program product. Background Art
[0002] The goal of hand-drawn sketch synthesis is to redraw the content image using the learned sketch style when given a content image. Existing sketch synthesis methods can be divided into pixel-based sketch synthesis and vector-based sketch synthesis.
[0003] Pixel-based sketch synthesis methods focus on generating pixel-based sketches using pixel graphics. Artbank (ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank) proposes to use global prompts through a diffusion model to learn style information from sketch images and transfer the learned style to arbitrary content images. UPDG (Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping) proposes to use local discriminators to learn detailed textures from sketch images. However, these methods always introduce some obvious artifacts and discordant patterns. They also do not support the generation of sketch images with different levels of abstraction. Pixel-based sketches contain individual pixels and may lose quality when scaled.
[0004] Vector-based sketch synthesis methods can generate sketches using scalable vector graphics without losing quality. Clipasso (CLIPasso Semantically-Aware Object Sketching) uses the CLIP model to learn local structure and global semantics from sketches and assigns gradients to a differentiable rasterizer to generate vector-based sketches. Although the above methods can generate vector-based sketches, they still have limitations in accurately describing the structure and semantics of the content image. Summary of the Invention
[0005] Based on this, it is necessary to provide a hand-drawn sketch synthesis method based on vector graphics to address the above technical problems.
[0006] This application provides a vector-based hand-drawn sketch synthesis method for synthesizing a hand-drawn sketch based on a content image, including:
[0007] Obtain a vector image based on the content image, and render a hand-drawn sketch based on the vector image;
[0008] superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;
[0009] obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into the diffusion model, and guiding the diffusion model to generate a natural image;
[0010] obtaining predicted noise in a process of generating the natural image using the diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;
[0011] Updating and optimizing the hand-drawn sketch by using a difference loss between the content image and the hand-drawn sketch;
[0012] The hand-drawn sketch is updated and optimized using the difference noise.
[0013] Optionally, obtaining a corresponding text description based on the content image, and performing edge detection on the content image to generate an edge map specifically includes: inputting the content image into an image-to-text model to obtain a corresponding text description, and performing Canny edge detection on the content image to generate an edge map;
[0014] Injecting the features of both the text description and the edge map into the diffusion model specifically includes: inputting the text description into a text extractor to extract text features, and injecting the text features into the diffusion model; inputting the edge map into a feature extraction network to extract edge features, and injecting the edge features into the diffusion model.
[0015] Optionally, injecting the text features into the diffusion model specifically includes: injecting the text features into the cross attention modules of both the encoder and the decoder in the diffusion model;
[0016] Injecting the edge features into the diffusion model specifically includes: injecting the edge features into a residual module of a decoder in the diffusion model.
[0017] Optionally, the difference noise is used to update and optimize the hand-drawn sketch, specifically including: using the difference noise to provide a gradient to a differentiable renderer to directly predict the update direction of the curve in the vector map, and the differentiable renderer is used to render the vector map into a hand-drawn sketch.
[0018] Optionally, the difference noise is used to update and optimize the hand-drawn sketch, specifically using the following formula:
[0019]
[0020] Where, represents the fractional distillation sampling loss, where φ represents the diffusion model, Represented by a differentiable renderer Rendering the hand-drawn sketch obtained, θ represents the initialization curve of the vector graph;
[0021] E t,∈ , which means seeking expectation;
[0022] w(t) is a hyperparameter representing the weight;
[0023] represents the difference noise, where represents the prediction noise, and ∈ represents the random noise.
[0024] Optionally, obtaining a vector map based on the content image specifically includes: extracting a content feature map based on the content image, converting the content feature map into a curve map, performing local sampling on the curve map to obtain an initialization curve, the initialization curve serving as the initial part of the vector map, and adjusting the degree of abstraction of the vector map and the hand-drawn sketch by changing the sampling accuracy of the local sampling.
[0025] Optionally, obtaining a content feature map based on the content image extraction specifically includes: extracting multi-scale features using different layers of a VGG model, and using an average feature map of the multi-scale features as the content feature map.
[0026] Optionally, the sampling accuracy of the local sampling is changed, specifically including: obtaining the control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and the initial set of control points constitutes different curve segments to constitute the initialization curve.
[0027] Optionally, sample some control points on some curves to obtain an initial set of control points with expected sampling accuracy, specifically including: obtaining the first control point on the target curve Control Point The target curve is centered on the target curve and the surrounding area is sampled according to the preset radius, and the control point Three adjacent control points Control point The set of is taken as the initial set of control points.
[0028] Optionally, the difference loss includes structural similarity loss and semantic similarity loss.
[0029] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the vector-based hand-drawn sketch synthesis method described in the present application.
[0030] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vector-based hand-drawn sketch synthesis method described in the present application are implemented.
[0031] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the vector-based hand-drawn sketch synthesis method described in the present application.
[0032] The hand-drawn sketch synthesis method based on vector graphics in this application has at least the following effects:
[0033] This application guides the diffusion model through text description and edge map, wherein the edge map can guide the content structure of the natural image, and the text description can guide the semantic structure of the natural image, so that the natural image can accurately describe the detailed information of the content image.
[0034] This application utilizes a stable diffusion model to output natural images. When the natural image is output stably, the predicted noise changes with the noisy image, that is, with the hand-drawn sketch. Therefore, the difference between the predicted noise and random noise can be used to update and optimize the hand-drawn sketch through gradients, improving the similarity of the hand-drawn sketch's content structure and semantic structure compared to the content image. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a flowchart of a method for synthesizing a hand-drawn sketch based on a vector diagram in one embodiment of the present application;
[0036] Figure 2 Schematic diagram of a model framework of a hand-drawn sketch synthesis method based on vector graphics in one embodiment of the present application;
[0037] Figure 3 This is a schematic diagram of a partial framework of a model of a hand-drawn sketch synthesis method based on a vector diagram in one embodiment of the present application;
[0038] Figure 4 This is a schematic diagram of a partial framework of a model of a hand-drawn sketch synthesis method based on a vector diagram in one embodiment of the present application;
[0039] Figure 5 Schematic diagram of a model framework of a hand-drawn sketch synthesis method based on vector graphics in one embodiment of the present application;
[0040] Figure 6FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0041] Current pixel-based sketch synthesis methods have made progress, but they can only support requirements at a specific level of abstraction and always introduce some obvious artifacts and discordant patterns. Although vector-based sketch synthesis methods can use structural similarity and semantic similarity losses to create vector-based sketches, they still cannot accurately describe the detailed information of the content image and have limitations in describing the structure and semantics of the content image.
[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0043] See also Figure 1 and Figure 2 An embodiment of the present application provides a method for synthesizing a hand-drawn sketch based on a vector image, which is used to synthesize a hand-drawn sketch according to a content image, including:
[0044] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;
[0045] Step S200 , superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;
[0046] Step S300: obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;
[0047] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;
[0048] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;
[0049] Step S600: using difference noise to update and optimize the hand-drawn sketch.
[0050] In various embodiments, the vector image in step S100 can be in SVG format, for example. The hand-drawn sketch is merely indicated to have an effect similar to a hand-drawn image. The hand-drawn sketch can be simply referred to as a sketch or a raster sketch. For step S200, reference can be made to the prior art regarding the use of a stable diffusion model. Step S300 guides the diffusion model using text descriptions and edge maps. The edge map can guide the content structure of the natural image, and the text description can guide the semantic structure of the natural image. Overall, the natural image can accurately describe the detailed information of the content image.
[0051] This embodiment uses the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch. When the hand-drawn sketch is updated, the noisy image is also updated, provided the random noise remains unchanged. However, due to the stable structure of the guided diffusion model, the natural image output by the diffusion model remains unchanged. The natural image is obtained by subtracting the predicted noise from the noisy image at each time step, based on the predictive power of the diffusion model.
[0052] At this time, the predicted noise actually predicted by the diffusion model can be obtained by subtracting the noisy image from the natural image. It can be understood that when the natural image is output stably, the predicted noise changes with the change of the noisy image, that is, the predicted noise changes with the change of the hand-drawn sketch. The random noise remains unchanged, so the difference noise between the predicted noise and the random noise can be used to update and optimize the hand-drawn sketch through the gradient. This embodiment freezes the structure of the diffusion model, making its output natural image stable and unchanged, and optimizes the input hand-drawn sketch with a fixed natural image, thereby improving the content structure and semantic structure similarity of the hand-drawn sketch compared to the content image.
[0053] In step S300, a corresponding text description is obtained based on the content image, and edge detection is performed on the content image to generate an edge map. This specifically includes: inputting the content image into a graph-to-text model to obtain the corresponding text description, and performing Canny edge detection on the content image to generate an edge map. The graph-to-text model can, for example, be a BLIP (Bootstrapping Language-Image Pre-training) model, which is used to extract text descriptions from the content image. As shown in the figure, the text description of the content image is "a meadow with mountains in the background."
[0054] This step generates a Canny edge map and corresponding text description based on the content image, improving the similarity between the natural image output by the diffusion model and the content image. During step S600, the differentiable rasterizer (differentiable renderer) can create a hand-drawn sketch based on the vector map, thereby more accurately describing the local structure of the content image.
[0055] In step S300, the features of both the text description and the edge map are injected into the diffusion model, specifically including: inputting the text description into a text extractor to extract text features, injecting the text features into the diffusion model, inputting the edge map into a feature extraction network to extract edge features, and injecting the edge features into the diffusion model.
[0056] Injecting text features into the diffusion model involves injecting them into the cross-attention modules of the encoder and decoder. Injecting edge features into the diffusion model involves injecting them into the residual module of the decoder. This step adds an edge map ψ as an edge and feeds it into the feature extraction network ControlNet to obtain edge features. This provides local structural information to guide and constrain the generation of natural images.
[0057] In some embodiments, a method for synthesizing hand-drawn sketches based on vector graphics is provided, which is used to explain and illustrate steps S100 to S600 of this application and provide corresponding optional sub-steps. The method includes: Step S1: Structural similarity loss and semantic similarity loss; Step S2: Stroke initialization based on the content feature map; and Step S3: Controllable fractional distillation sampling loss.
[0058] Step S1, structural similarity loss and semantic similarity loss, corresponds to the optional sub-steps of step S500.
[0059] In step S500 , the difference loss includes structural similarity loss and semantic similarity loss.
[0060] The structural similarity loss can be calculated as follows:
[0061]
[0062] Where, Represents the structural similarity loss, l represents the accumulated lth layer, and Norm represents the mean variance channel normalization. i Represents the i-th layer of the VGG model, where i can be 4 and 5, for example, to obtain the 4th and 5th layer outputs Relu4_1 and Relu5_1 of the VGG model respectively. Indicates hand-drawn sketch, I c Represents a content image.
[0063] The semantic similarity loss can be calculated as follows:
[0064]
[0065] Where, represents semantic similarity loss, l represents the accumulated lth layer, CLIP irepresents the i-th layer of the visual encoder (CLIP model) that calculates semantic similarity, Indicates hand-drawn sketch, I c Represents a content image.
[0066] For a given content image, step S500 ensures that the generated hand-drawn sketch is faithful to the semantics and structure of the content image.
[0067] Step S2, stroke initialization based on the content feature map, corresponds to an optional sub-step of step S100.
[0068] If the starting point of each curve in the vector map is randomly initialized, the synthesis process may be very time-consuming. In addition, the highly non-convex structural similarity and semantic similarity loss equations make the synthesized raster sketch susceptible to random initialization, and the local structure and global semantics of the synthesized sketch will be severely degraded. To this end, this embodiment introduces a novel content feature map-based stroke initialization, named "MFSI". Specifically, the hand-drawn sketch is converted into a set of Bezier curves, and the initialization curve is sampled to obtain the initialization curve, and then the initialization curve is optimized through a differentiable rasterizer.
[0069] See also Figure 3 and Figure 4 In step S100, a vector map is obtained based on the content image. This specifically includes: extracting a content feature map from the content image, converting the content feature map into a curve map, and performing local sampling on the curve map to obtain an initialization curve. The initialization curve serves as the initial portion of the vector map. The degree of abstraction of the vector map and the hand-drawn sketch is adjusted by varying the sampling accuracy of the local sampling. This step can change the degree of abstraction of the hand-drawn sketch and accelerate the optimization of curve parameters in the vector map.
[0070] The content feature map is obtained based on the content image extraction, specifically including: extracting multi-scale features using different layers of the VGG model, and taking the average feature map of the multi-scale features as the content feature map. Specifically, the pre-trained VGG model is used to extract the content feature map from the content image I c Multi-scale features are extracted from the VGG model, including the multi-scale features output by layers 3, 4, and 5, respectively, which are denoted as Relu3_1, Relu4_1, and Relu5_1. The average feature map of the three is then calculated to obtain a fused feature map, which is used as the content feature map.
[0071] The content feature map is converted into a curve map, specifically including normalizing the content feature map using a softmax function to obtain a dot matrix composed of data points, and then obtaining a curve map (or distribution map) representing the distribution of the curve.
[0072] Changing the sampling accuracy of local sampling specifically includes: obtaining control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and composing different curve segments with the initial set of control points to form an initialization curve as shown in the figure.
[0073] Sampling some control points on some curves to obtain an initial set of control points with expected sampling accuracy, specifically including: obtaining the first control point on the target curve Control Point The target curve is centered and samples the surrounding area according to the preset radius, Three adjacent control points Control point The set of is taken as the initial set of control points.
[0074] The number of curves used for sampling in the graph, as well as the number of control points used for sampling each curve, can be adjusted to change the degree of abstraction in the sketch. As you can see, the fewer curves used for sampling, the more abstract the resulting sketch; conversely, the more realistic the sketch.
[0075] For any target curve, the first control point of the target curve Then the control point The other three control points within a preset radius (the radius can be 0.05 of the image size, for example) are sampled to define the initial set. The control point can be understood as the data point used by the computer to record the curve. For each curve, the first control point Then sample n positions of The other three control points within the preset radius Sampling is performed to define the initial set
[0076] Step S3, controllable fractional distillation sampling loss, corresponds to an optional sub-step of step S600.
[0077] Given only a rough text description, natural images cannot reflect the detailed structure of the input content image. This step introduces a novel controllable fractional distillation sampling loss to further learn the local structure and global semantics from the content image, and provides gradients to the differentiable renderer to generate hand-drawn sketches based on vector graphics to more accurately describe the structure and semantics of the content image. This step names the controllable fractional distillation sampling loss "CSDS", which encourages the synthesized hand-drawn sketches to better match the text description and edge map. Using the fractional distillation sampling loss (SDS) for simplification still fails to achieve more accurate results.
[0078] Specifically, step S600 uses difference noise to update and optimize the hand-drawn sketch, which specifically includes: using the difference noise to provide a gradient to the differentiable renderer to directly predict the update direction of the curve in the vector map, and the differentiable renderer is used to render the vector map into a hand-drawn sketch. The above process is specifically performed using the following formula:
[0079]
[0080] Where, represents the fractional distillation sampling loss, where φ represents the diffusion model, Represented by a differentiable renderer The hand-drawn sketch obtained by rendering, θ represents the initialization curve of the vector graph;
[0081] E t,∈ , which means seeking expectation;
[0082] w(t) is a hyperparameter representing weight. Different weight constant values can be set accordingly for different values of t.
[0083] represents the difference noise, where represents prediction noise, and ∈ represents random noise.
[0084] t~U(0.05,0.95) is a hyperparameter, which is magnified by 1000 times when the fractional distillation sampling loss is applied, and is used to indicate that the time step t is uniformly distributed between 50 and 950.
[0085] This step directly predicts the update direction of the curve by providing gradients to the differentiable renderer. Specifically, given a raster sketch (3 channels H×W pixels), the hand-drawn sketch is first encoded into a latent representation using a variational autoencoder (VAE encoder) in f represents the encoder downsampling factor. The input is then denoised using random noise in the diffusion model, and then denoised using the diffusion model. The diffusion model uses the edge map of the content image during denoising. Therefore, the diffusion model can pass the edge map's details to the differentiable renderer in the form of gradients, enabling it to better learn the details of the content image.
[0086] See also Figure 5 The hand-drawn sketch synthesis method based on vector graphics in each embodiment of this application can generate high-quality hand-drawn sketch synthesis, so that the hand-drawn sketch can accurately describe the structure and semantics of the content image. The framework that combines steps S1 to S3 as a whole is named "VectorSketcher", which can better accurately describe the local structure and global semantic information of the content image. Summarizing all the above losses and obtaining the final target loss:
[0087]
[0088] Where, represents the total loss, represents the structural similarity loss, represents the semantic similarity loss, represents the controllable fractional distillation sampling loss, λ1, λ2, and λ3 are weight coefficients, for example, all of which are set to 1.
[0089] The framework of each embodiment of the present application is used to draw a hand-drawn sketch based on a vector map according to an arbitrary content image, and the hand-drawn sketch obtained by drawing can accurately describe the content and semantics of the content image. Step S2, based on the stroke initialization of the content feature map, can use the content feature map to obtain the initialization curve, accelerate the optimization process of the hand-drawn sketch, and promote the accuracy of the synthesized hand-drawn sketch to describe the content image. Specifically, in the process of obtaining the initialization curve, the sampling accuracy of the local sampling is changed to adjust the degree of abstraction of the vector map and the hand-drawn sketch, thereby obtaining a hand-drawn sketch with multiple levels of abstraction based on the vector map. Step S3, controllable fractional distillation sampling loss, can provide a gradient to the differentiable renderer, promote the update and optimization of the hand-drawn sketch based on the vector map, so that it can more accurately describe the content structure and semantic information of the content image.
[0090] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0091] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a hand-drawn sketch synthesis method based on a vector diagram is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.
[0092] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0093] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;
[0094] Step S200 , superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;
[0095] Step S300: obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;
[0096] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;
[0097] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;
[0098] Step S600: using difference noise to update and optimize the hand-drawn sketch.
[0099] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:
[0100] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;
[0101] Step S200 , superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;
[0102] Step S300: obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;
[0103] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;
[0104] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;
[0105] Step S600: using difference noise to update and optimize the hand-drawn sketch.
[0106] In this embodiment, the computer program product includes a program code portion for executing the steps of the vector-based hand-drawn sketch synthesis method in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for download with the aid of a hardware description language.
[0107] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0108] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.
[0109] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A hand-drawn sketch synthesis method based on vector graphics, which is used to synthesize hand-drawn sketches according to content images, characterized in that: include: Obtain a vector image based on the content image, and render a hand-drawn sketch based on the vector image; superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model; obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into the diffusion model, and guiding the diffusion model to generate a natural image; obtaining predicted noise in a process of generating the natural image using the diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise; Updating and optimizing the hand-drawn sketch by using a difference loss between the content image and the hand-drawn sketch; The difference noise is used to update and optimize the hand-drawn sketch, specifically including: using the difference noise to provide a gradient to a differentiable renderer to directly predict the update direction of the curve in the vector map, and the differentiable renderer is used to render the vector map into a hand-drawn sketch, specifically using the following formula: Where, represents the fractional distillation sampling loss, where represents the diffusion model, Represented by a differentiable renderer Rendering the obtained hand-drawn sketch, An initialization curve representing the vector diagram; , which means seeking expectation; , is a hyperparameter, representing the weight; , represents the difference noise, where represents the prediction noise, represents the random noise.
2. The hand-drawn sketch synthesis method according to claim 1, wherein: Obtaining a corresponding text description based on the content image, and performing edge detection on the content image to generate an edge map, specifically comprising: inputting the content image into an image-to-text model to obtain a corresponding text description, and performing Canny edge detection on the content image to generate an edge map; Injecting the features of both the text description and the edge map into the diffusion model specifically includes: inputting the text description into a text extractor to extract text features, and injecting the text features into the diffusion model; inputting the edge map into a feature extraction network to extract edge features, and injecting the edge features into the diffusion model.
3. The hand-drawn sketch synthesis method according to claim 1, wherein: Obtaining a vector graph based on the content image specifically includes: A content feature map is obtained based on the content image extraction, the content feature map is converted into a curve map, and the curve map is locally sampled to obtain an initialization curve. The initialization curve serves as the initial part of the vector map, and the degree of abstraction of the vector map and the hand-drawn sketch is adjusted by changing the sampling accuracy of the local sampling.
4. The hand-drawn sketch synthesis method according to claim 3, wherein: Changing the sampling accuracy of the local sampling specifically includes: obtaining control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and the initial set of control points is composed of different curve segments to constitute the initialization curve.
5. The hand-drawn sketch synthesis method according to claim 4, wherein: Sampling some control points on some curves to obtain an initial set of control points with expected sampling accuracy, specifically including: obtaining the first control point on the target curve , with control points The target curve is centered on the target curve and the surrounding area is sampled according to the preset radius, and the control point Three adjacent control points { , , }, the control point { , , , } is used as the initial set of control points.
6. The hand-drawn sketch synthesis method according to claim 1, wherein: The difference loss includes structural similarity loss and semantic similarity loss.
7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the vector-based hand-drawn sketch synthesis method according to any one of claims 1 to 6.
8. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the vector-based hand-drawn sketch synthesis method according to any one of claims 1 to 6 are implemented.