Vector diagram-based freehand sketch synthesis method, computer equipment and program product

Abstract

The invention relates to a vector diagram-based freehand sketch synthesis method, computer equipment and program product, and the method comprises the steps: obtaining a vector diagram based on a content image, and obtaining a freehand sketch based on vector diagram rendering; superposing the random noise to the freehand sketch to obtain a noisy image, and inputting the noisy image to a diffusion model; obtaining corresponding text description based on the content image, performing edge detection on the content image to generate an edge image, injecting features of the text description and the edge image into a diffusion model, and guiding the diffusion model to generate a natural image; according to the natural image and the noisy image, obtaining prediction noise in the process that the diffusion model generates the natural image, and according to the prediction noise and the random noise, obtaining difference noise; updating and optimizing the freehand sketch by using the difference loss of the content image and the freehand sketch; and updating and optimizing the freehand sketch by using the difference noise. According to the method, the similarity of the content structure and the semantic structure of the freehand sketch relative to the content image is improved.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a vector-based hand-drawn sketch synthesis method, computer device, and program product. Background Art

[0002] The goal of hand-drawn sketch synthesis is to redraw the content image using the learned sketch style when given a content image. Existing sketch synthesis methods can be divided into pixel-based sketch synthesis and vector-based sketch synthesis.

[0003] Pixel-based sketch synthesis methods focus on generating pixel-based sketches using pixel graphics. Artbank (ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank) proposes to use global prompts through a diffusion model to learn style information from sketch images and transfer the learned style to arbitrary content images. UPDG (Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping) proposes to use local discriminators to learn detailed textures from sketch images. However, these methods always introduce some obvious artifacts and discordant patterns. They also do not support the generation of sketch images with different levels of abstraction. Pixel-based sketches contain individual pixels and may lose quality when scaled.

[0004] Vector-based sketch synthesis methods can generate sketches using scalable vector graphics without losing quality. Clipasso (CLIPasso Semantically-Aware Object Sketching) uses the CLIP model to learn local structure and global semantics from sketches, and assigns gradients to a differentiable rasterizer to generate vector-based sketches. Although the above methods can generate vector-based sketches, they still have limitations in accurately describing the structure and semantics of the content image. Summary of the invention

[0005] Based on this, it is necessary to provide a hand-drawn sketch synthesis method based on vector graphics to address the above technical problems.

[0006] The present application provides a hand-drawn sketch synthesis method based on a vector diagram, which is used to synthesize a hand-drawn sketch according to a content image, including:

[0007] Obtain a vector diagram based on the content image, and obtain a hand-drawn sketch based on the vector diagram rendering;

[0008] superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;

[0009] Obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into the diffusion model, and guiding the diffusion model to generate a natural image;

[0010] Obtaining predicted noise in a process of generating the natural image using the diffusion model according to the natural image and the noisy image, and obtaining differential noise according to the predicted noise and the random noise;

[0011] Using the difference loss between the content image and the hand-drawn sketch, updating and optimizing the hand-drawn sketch;

[0012] The hand-drawn sketch is updated and optimized by utilizing the difference noise.

[0013] Optionally, obtaining a corresponding text description based on the content image, and performing edge detection on the content image to generate an edge map specifically includes: inputting the content image into an image-to-text model to obtain a corresponding text description, and performing Canny edge detection on the content image to generate an edge map;

[0014] Injecting the features of both the text description and the edge map into the diffusion model specifically includes: inputting the text description into a text extractor to extract text features, and injecting the text features into the diffusion model; inputting the edge map into a feature extraction network to extract edge features, and injecting the edge features into the diffusion model.

[0015] Optionally, injecting the text feature into the diffusion model specifically includes: injecting the text feature into a cross attention module of both an encoder and a decoder in the diffusion model;

[0016] Injecting the edge feature into the diffusion model specifically includes: injecting the edge feature into a residual module of a decoder in the diffusion model.

[0017] Optionally, the difference noise is used to update and optimize the hand-drawn sketch, specifically including: using the difference noise to provide a gradient to a differentiable renderer to directly predict an update direction of a curve in the vector map, the differentiable renderer being used to render the vector map into a hand-drawn sketch.

[0018] Optionally, the difference noise is used to update and optimize the hand-drawn sketch, specifically using the following formula:

[0019]

[0020] In the formula, represents the fractional distillation sampling loss, where φ represents the diffusion model, Represents a differentiable renderer Rendering the hand-drawn sketch obtained, θ represents an initialization curve of the vector graph;

[0021] E t,∈ , which means seeking expectation;

[0022] w(t) is a hyperparameter, representing the weight;

[0023] represents the difference noise, where represents the prediction noise, ∈ represents the random noise.

[0024] Optionally, obtaining a vector map based on the content image specifically includes: obtaining a content feature map based on the content image extraction, converting the content feature map into a curve graph, performing local sampling on the curve graph to obtain an initialization curve, the initialization curve serving as an initial part of the vector map, and adjusting the degree of abstraction of the vector map and the hand-drawn sketch by changing the sampling accuracy of the local sampling.

[0025] Optionally, obtaining a content feature map based on the content image extraction specifically includes: extracting multi-scale features using different layers of a VGG model, and using an average feature map of the multi-scale features as the content feature map.

[0026] Optionally, the sampling accuracy of the local sampling is changed, specifically comprising: obtaining control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and the initial set of control points constitutes different curve segments to constitute the initialization curve.

[0027] Optionally, some control points on some curves are sampled to obtain an initial set of control points with expected sampling accuracy, specifically including: obtaining the first control point on the target curve By control point The control point is sampled on the target curve according to the preset radius and the surrounding area is sampled. Three adjacent control points Control Point The set of is taken as the initial set of control points.

[0028] Optionally, the difference loss includes structural similarity loss and semantic similarity loss.

[0029] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the vector-based hand-drawn sketch synthesis method described in the present application.

[0030] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the vector-image-based hand-drawn sketch synthesis method described in the present application.

[0031] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the vector-image-based hand-drawn sketch synthesis method described in the present application.

[0032] The hand-drawn sketch synthesis method based on vector graphics in this application has at least the following effects:

[0033] The present application guides the diffusion model through text description and edge map, wherein the edge map can guide the content structure of the natural image, and the text description can guide the semantic structure of the natural image, so that the natural image can accurately describe the detailed information of the content image.

[0034] This application uses a stable diffusion model to output natural images. When the natural image is output stably, the prediction noise changes with the change of the noisy image, that is, the prediction noise changes with the change of the hand-drawn sketch. Therefore, the difference noise between the prediction noise and the random noise can be used to update and optimize the hand-drawn sketch through the gradient, which improves the similarity of the content structure and semantic structure of the hand-drawn sketch compared to the content image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for synthesizing a hand-drawn sketch based on a vector diagram in one embodiment of the present application;

[0036] Figure 2 A schematic diagram of a model framework of a hand-drawn sketch synthesis method based on a vector diagram in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a partial framework of a model of a hand-drawn sketch synthesis method based on a vector diagram in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of a partial framework of a model of a hand-drawn sketch synthesis method based on a vector diagram in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a model framework of a hand-drawn sketch synthesis method based on a vector diagram in an embodiment of the present application;

[0040] Figure 6FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] At present, pixel-based sketch synthesis methods have made progress, but they can only support requirements at a specific level of abstraction and always introduce some obvious artifacts and discordant patterns. Although vector-based sketch synthesis methods can use structural similarity and semantic similarity losses to create vector-based sketches, they still cannot accurately describe the details of the content image and have limitations in describing the structure and semantics of the content image.

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] See also Figure 1 and Figure 2 An embodiment of the present application provides a method for synthesizing a hand-drawn sketch based on a vector image, which is used to synthesize a hand-drawn sketch according to a content image, including:

[0044] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;

[0045] Step S200, superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;

[0046] Step S300, obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;

[0047] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;

[0048] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;

[0049] Step S600, using difference noise to update and optimize the hand-drawn sketch.

[0050] In each embodiment, the vector image in step S100 may be in SVG format, for example. The hand-drawn sketch is only indicated to have an effect similar to that of a hand-drawn picture. The hand-drawn sketch may be referred to as a sketch or a raster sketch. For step S200, reference may be made to the use of a stable diffusion model in the prior art. Step S300 guides the diffusion model through text description and edge graph, wherein the edge graph can guide the content structure of the natural image, and the text description can guide the semantic structure of the natural image, so that the natural image can accurately describe the detailed information of the content image as a whole.

[0051] This embodiment uses the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch. If the hand-drawn sketch is updated, the noisy image is updated when the random noise remains unchanged. Since the structure of the guided diffusion model is stable, the natural image output by the diffusion model remains unchanged. The acquisition of the natural image is based on the prediction ability of the diffusion model, which is obtained by subtracting the prediction noise from the noisy image at a time step.

[0052] At this time, the prediction noise actually predicted by the diffusion model can be obtained by subtracting the noisy image from the natural image. It can be understood that when the natural image is stably output, the prediction noise changes with the change of the noisy image, that is, the prediction noise changes with the change of the hand-drawn sketch. The random noise remains unchanged, so the difference noise between the prediction noise and the random noise can be used to update and optimize the hand-drawn sketch through the gradient. This embodiment freezes the structure of the diffusion model, so that the natural image output by it is stable and unchanged, and optimizes the input hand-drawn sketch with a fixed natural image, thereby improving the content structure and semantic structure similarity of the hand-drawn sketch compared to the content image.

[0053] In step S300, a corresponding text description is obtained based on the content image, and edge detection is performed on the content image to generate an edge map, specifically including: inputting the content image into a graph-to-text model to obtain a corresponding text description, and performing Canny edge detection on the content image to generate an edge map. The graph-to-text model can be, for example, a BLIP (Bootstrapping Language-Image Pre-training) guided language image pre-training model, which is used to extract text descriptions from content images. The text description of the content image shown in the figure is "a meadow with mountains in the background".

[0054] This step generates a Canny edge map and a corresponding text description based on the content image to improve the similarity between the natural image output by the diffusion model and the content image. In the process of executing step S600, the differentiable rasterizer (differentiable renderer) can create a hand-drawn sketch based on a vector image, thereby more accurately describing the local structure of the content image.

[0055] In step S300, the features of both the text description and the edge map are injected into the diffusion model, specifically including: inputting the text description into the text extractor to extract text features, injecting the text features into the diffusion model, inputting the edge map into the feature extraction network to extract edge features, and injecting the edge features into the diffusion model.

[0056] Injecting text features into the diffusion model specifically includes: injecting text features into the cross attention module of the encoder and the decoder in the diffusion model. Injecting edge features into the diffusion model specifically includes: injecting edge features into the residual module of the decoder in the diffusion model. In this step, the edge map ψ is added as an edge, and the edge map ψ is input into the feature extraction network ControlNet to obtain edge features to provide local structural information and guide and constrain the generation of natural images.

[0057] In some embodiments, a method for synthesizing hand-drawn sketches based on vector graphics is provided, which is used to correspondingly explain and illustrate steps S100 to S600 of the present application and provide corresponding optional sub-steps. Including: step S1, structural similarity loss and semantic similarity loss. Step S2, stroke initialization based on content feature graph. Step S3, controllable fractional distillation sampling loss.

[0058] Step S1, structural similarity loss and semantic similarity loss, corresponds to the optional sub-steps of step S500.

[0059] In step S500 , the difference loss includes structural similarity loss and semantic similarity loss.

[0060] The structural similarity loss can be calculated as follows:

[0061]

[0062] In the formula, Represents the structural similarity loss, l represents the accumulated lth layer, and Norm represents the mean variance channel normalization. i Represents the i-th layer of the VGG model, where i can be 4 and 5, for example, corresponding to the 4th and 5th layer outputs Relu4_1 and Relu5_1 of the VGG model respectively. Indicates a hand-drawn sketch, I c Represents a content image.

[0063] The semantic similarity loss can be calculated as follows:

[0064]

[0065] In the formula, represents the semantic similarity loss, l represents the accumulated lth layer, CLIP irepresents the i-th layer of the visual encoder (CLIP model) that calculates semantic similarity, Indicates a hand-drawn sketch, I c Represents a content image.

[0066] For a given content image, step S500 ensures that the generated hand-drawn sketch is faithful to the semantics and structure of the content image.

[0067] Step S2, initialization of strokes based on the content feature map, corresponds to an optional sub-step of step S100.

[0068] If the starting point of each curve in the vector map is randomly initialized, the synthesis process may be very time-consuming. In addition, the highly non-convex structural similarity and semantic similarity loss equations cause the synthesized raster sketch to be susceptible to random initialization, and the local structure and global semantics of the synthesized sketch will be severely degraded. For this reason, this embodiment introduces a novel content feature map-based stroke initialization, named "MFSI". Specifically, the hand-drawn sketch is converted into a set of Bezier curves, and the initialization curve is sampled to obtain the initialization curve, and then the initialization curve is optimized through a differentiable rasterizer.

[0069] See also Figure 3 and Figure 4 In step S100, a vector map is obtained based on the content image, specifically including: extracting a content feature map based on the content image, converting the content feature map into a curve map, performing local sampling on the curve map to obtain an initialization curve, the initialization curve is used as the initial part of the vector map, and the degree of abstraction of the vector map and the hand-drawn sketch is adjusted by changing the sampling accuracy of the local sampling. This step can change the degree of abstraction of the hand-drawn sketch and accelerate the optimization of the curve parameters in the vector map.

[0070] The content feature map is obtained based on the content image extraction, specifically including: extracting multi-scale features using different layers of the VGG model, and taking the average feature map of the multi-scale features as the content feature map. Specifically, the pre-trained VGG model is used to extract the content feature map from the content image I c The multi-scale features are extracted from the 3rd, 4th and 5th layers of the VGG model, which are respectively denoted as Relu3_1, Relu4_1 and Relu5_1. Then the average feature map of the three is calculated to obtain the fused feature map, which is used as the content feature map.

[0071] The content feature map is converted into a curve map, specifically including normalizing the content feature map using a softmax function to obtain a dot matrix map composed of data points, and then obtaining a curve map (or distribution map) representing the distribution of the curve.

[0072] Changing the sampling accuracy of local sampling specifically includes: obtaining control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and the initial set of control points composes different curve segments to form an initialization curve as shown in the figure.

[0073] Sampling some control points on some curves to obtain an initial set of control points with expected sampling accuracy, including: obtaining the first control point on the target curve By control point The target curve is centered and samples the surrounding area according to the preset radius, which is consistent with the control point Three adjacent control points Control Point The set of is taken as the initial set of control points.

[0074] The number of curves used for sampling in the curve graph and the number of control points for sampling each curve can be manually adjusted to change the abstraction level of the hand-drawn sketch. It can be understood that the fewer the number of curves used for sampling, the more abstract the final hand-drawn sketch will be; conversely, the more realistic the hand-drawn sketch will be.

[0075] For any target curve, the first control point of the target curve Then the control point The other three control points within a preset radius (the radius can be, for example, 0.05 of the image size) are sampled to define the initial set. The control point can be understood as a data point used by the computer to record the curve. For each curve, the first control point Then, The other three control points that are smaller than the preset radius Sampling is performed to define the initial set

[0076] Step S3, controllable fractional distillation sampling loss, corresponds to an optional sub-step of step S600.

[0077] Given only a rough text description, natural images cannot reflect the detailed structure of the input content image. This step introduces a novel controllable fractional distillation sampling loss to further learn local structure and global semantics from the content image, and provide gradients to the differentiable renderer to generate hand-drawn sketches based on vector graphics to more accurately describe the structure and semantics of the content image. This step names the controllable fractional distillation sampling loss "CSDS", which encourages the synthesized hand-drawn sketches to better match the text description and edge map. Simplifying using the fractional distillation sampling loss (SDS) still does not achieve more accurate results.

[0078] Specifically, step S600, using difference noise to update and optimize the hand-drawn sketch, specifically includes: using difference noise to provide a gradient to a differentiable renderer to directly predict the update direction of the curve in the vector map, and the differentiable renderer is used to render the vector map into a hand-drawn sketch. The above process is specifically performed using the following formula:

[0079]

[0080] In the formula, represents the fractional distillation sampling loss, where φ represents the diffusion model, Represents a differentiable renderer The hand-drawn sketch obtained by rendering, θ represents the initialization curve of the vector graph;

[0081] E t,∈ , which means seeking expectation;

[0082] w(t) is a hyperparameter representing weight. Different weight constant values ​​can be set accordingly for different values ​​of t.

[0083] represents the difference noise, where represents prediction noise, and ∈ represents random noise.

[0084] t~U(0.05,0.95) is a hyperparameter, which is magnified 1000 times when the fractional distillation sampling loss is applied, and is used to indicate that the time step t is uniformly distributed between 50 and 950.

[0085] This step directly predicts the update direction of the curve by providing gradients to the differentiable renderer. Specifically, given a raster sketch (3 channels H×W pixels), the hand-drawn sketch is first encoded into a latent representation using a variational autoencoder (VAE encoder) in f represents the encoder downsampling factor. Then, random noise is used to add noise to the input diffusion model, and the diffusion model is used for denoising. The diffusion model uses the edge map of the content image in the denoising process, so the diffusion model can pass the detailed information of the edge map to the differentiable renderer in the form of gradients, so that it can generate better detailed information of the content image.

[0086] See also Figure 5 , the hand-drawn sketch synthesis method based on vector graphics in each embodiment of the present application can realize the generation of high-quality hand-drawn sketch synthesis, so that the hand-drawn sketch can accurately describe the structure and semantics of the content image. The framework that combines steps S1 to S3 as a whole is named "VectorSketcher", which can better accurately describe the local structure and global semantic information of the content image. Summarize all the above losses and get the final target loss:

[0087]

[0088] In the formula, represents the total loss, represents the structural similarity loss, represents the semantic similarity loss, represents the controllable fractional distillation sampling loss, λ1, λ2, λ3 are weight coefficients, for example, all of which are 1.

[0089] The framework of each embodiment of the present application is used to draw a hand-drawn sketch based on a vector map according to an arbitrary content image, and the hand-drawn sketch obtained by drawing can accurately describe the content and semantics of the content image. Step S2, stroke initialization based on the content feature map, can use the content feature map to obtain an initialization curve, accelerate the optimization process of the hand-drawn sketch, and promote the accuracy of the synthesized hand-drawn sketch to describe the content image. Specifically, in the process of obtaining the initialization curve, the sampling accuracy of the local sampling is changed to adjust the degree of abstraction of the vector map and the hand-drawn sketch, so as to obtain a hand-drawn sketch with multiple levels of abstraction based on the vector map. Step S3, controllable fractional distillation sampling loss, can provide gradients to the differentiable renderer, promote the update and optimization of the hand-drawn sketch based on the vector map, so that it can more accurately describe the content structure and semantic information of the content image.

[0090] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a hand-drawn sketch synthesis method based on a vector diagram is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0092] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0093] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;

[0094] Step S200, superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;

[0095] Step S300, obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;

[0096] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;

[0097] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;

[0098] Step S600, using difference noise to update and optimize the hand-drawn sketch.

[0099] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0100] Step S100, obtaining a vector image based on the content image, and obtaining a hand-drawn sketch based on the vector image rendering;

[0101] Step S200, superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model;

[0102] Step S300, obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into a diffusion model, and guiding the diffusion model to generate a natural image;

[0103] Step S400, obtaining predicted noise in the process of generating the natural image using a diffusion model according to the natural image and the noisy image, and obtaining difference noise according to the predicted noise and the random noise;

[0104] Step S500, using the difference loss between the content image and the hand-drawn sketch to update and optimize the hand-drawn sketch;

[0105] Step S600, using difference noise to update and optimize the hand-drawn sketch.

[0106] In the present embodiment, the computer program product includes a program code portion for executing the steps of the hand-drawn sketch synthesis method based on the vector image in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or functionality may be provided for download by means of a hardware description language.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0108] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0109] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A hand-drawn sketch synthesis method based on vector graphics, used to synthesize hand-drawn sketches according to content images, characterized in that: include: Obtain a vector diagram based on the content image, and obtain a hand-drawn sketch based on the vector diagram rendering; superimposing random noise onto the hand-drawn sketch to obtain a noisy image, and inputting the noisy image into a diffusion model; Obtaining a corresponding text description based on the content image, performing edge detection on the content image to generate an edge map, injecting features of both the text description and the edge map into the diffusion model, and guiding the diffusion model to generate a natural image; Obtaining predicted noise in a process of generating the natural image using the diffusion model according to the natural image and the noisy image, and obtaining differential noise according to the predicted noise and the random noise; Using the difference loss between the content image and the hand-drawn sketch, updating and optimizing the hand-drawn sketch; The hand-drawn sketch is updated and optimized by utilizing the difference noise.

2. The hand-drawn sketch synthesis method according to claim 1, characterized in that: Obtaining a corresponding text description based on the content image, and performing edge detection on the content image to generate an edge map, specifically comprising: inputting the content image into an image-to-text model to obtain a corresponding text description, and performing Canny edge detection on the content image to generate an edge map; Injecting the features of both the text description and the edge map into the diffusion model specifically includes: inputting the text description into a text extractor to extract text features, and injecting the text features into the diffusion model; inputting the edge map into a feature extraction network to extract edge features, and injecting the edge features into the diffusion model.

3. The hand-drawn sketch synthesis method according to claim 1, characterized in that: The method uses the difference noise to update and optimize the hand-drawn sketch, specifically comprising: using the difference noise to provide a gradient to a differentiable renderer to directly predict the update direction of the curve in the vector map, and the differentiable renderer is used to render the vector map into a hand-drawn sketch.

4. The hand-drawn sketch synthesis method according to claim 3, characterized in that: The hand-drawn sketch is updated and optimized by using the difference noise, specifically using the following formula: In the formula, represents the fractional distillation sampling loss, where φ represents the diffusion model, Represents a differentiable renderer Rendering the hand-drawn sketch obtained, θ represents an initialization curve of the vector graph; E t，∈ , which means seeking expectation; w(t) is a hyperparameter, representing the weight; represents the difference noise, where represents the prediction noise, ∈ represents the random noise.

5. The hand-drawn sketch synthesis method according to claim 1, characterized in that: Obtaining a vector graph based on the content image specifically includes: A content feature map is obtained based on the content image extraction, the content feature map is converted into a curve graph, and the curve graph is locally sampled to obtain an initialization curve. The initialization curve serves as the initial part of the vector map, and the degree of abstraction of the vector map and the hand-drawn sketch is adjusted by changing the sampling accuracy of the local sampling.

6. The hand-drawn sketch synthesis method according to claim 5, characterized in that: Changing the sampling accuracy of the local sampling specifically includes: obtaining control points of each curve in the curve graph, sampling some control points on some curves, obtaining an initial set of control points with expected sampling accuracy, and the initial set of control points constitutes different curve segments to form the initialization curve.

7. The hand-drawn sketch synthesis method according to claim 6, characterized in that: Sampling some control points on some curves to obtain an initial set of control points with expected sampling accuracy, including: obtaining the first control point on the target curve By control point The control point is sampled on the target curve according to the preset radius and the surrounding area is sampled. Three adjacent control points Control Point The set of is taken as the initial set of control points.

8. The hand-drawn sketch synthesis method according to claim 1, characterized in that: The difference loss includes structural similarity loss and semantic similarity loss.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the vector-based hand-drawn sketch synthesis method according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the vector-based hand-drawn sketch synthesis method as described in any one of claims 1 to 8 are implemented.

Images