Semantic layout driving-based scene freehand sketch generation and editing method, system and equipment and medium

Abstract

A scene freehand sketch generation and editing method, system and device based on semantic layout driving and a medium, comprising: preprocessing a scene image to obtain an image sketch image pair of a single object in the preprocessed scene image; constructing a scene image panoramic segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module and a scene sketch combination module; performing model training on the scene layout modeling module and the single object sketch generation module; constructing a complete scene sketch generation model; the system, the equipment and the medium are used for implementing the method. According to the method, a plurality of corresponding human hand-drawn scene sketches with different scene layouts, various object forms and editable sketch details can be generated according to the input scene image, so that the finally generated scene sketches have scene-level hand-drawn abstract characteristics, and semantic characteristics of scene objects are highlighted.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary technical field of computer vision and computer graphics, and specifically relates to a method, system, device and medium for generating and editing a scene hand-drawn sketch driven by semantic layout. Background Art

[0002] The scene sketch generation task is an important research direction in the field of computer vision. Scene sketch-related applications are also very extensive. For example, hand-drawn scene sketch modeling of 3D scenes can help architectural designers analyze, design and display their results; in the field of games, through the imagined scene sketches, game scene designers can quickly build exquisite game scenes according to the scene sketches, which can greatly reduce the cost of game development and improve development efficiency. Therefore, the generation of hand-drawn sketches of scenes is a very promising research topic. But at the same time, scene sketch generation is also a very challenging task. A scene may contain various cross-relationships between foreground and background, such as a single object with a simple background, a single object with a complex background, and multiple objects with a complex background. Therefore, it is very difficult for computers to understand this complex scene relationship and generate reasonable sketches.

[0003] Existing sketch-related work is mainly based on deep learning methods, using convolutional neural networks to extract image features of original images and sketches. Compared with traditional sketch generation algorithms, these methods can generate sketches of higher quality. However, existing methods also have some limitations. For example, sketch generation only focuses on a single object and does not expand the generation task to the scene level. At the technical level, most existing methods use edge detection methods to extract edge information of the original image, resulting in the generated sketch visually conveying more geometric features of the object and failing to highlight semantic features.

[0004] The patent application document with publication number CN106126581B discloses a hand-drawn sketch image retrieval method based on deep learning. The invention realizes the conversion of color images to sketch-like images through edge contour detection technology and non-maximum suppression technology, and then uses deep learning technology to learn the deep feature distinguishing feature representation of the queried sketch and the sketch-like image, and finally uses the learned deep features to perform sketch-based image retrieval. However, the image converted by this method only contains a single foreground object, resulting in the generated sketch-like image only generating a sketch of a single object, so it does not generate a sketch at the level of the entire scene; in addition, due to the use of edge detection technology, the generated sketch-like image only reflects the geometric edge features of the object, while ignoring the semantic features of the object, and the generated sketch does not have the characteristics of human hand-drawing.

[0005] The patent application document with publication number CN108009286A discloses a sketch retrieval method based on deep learning. The invention uses a method of calculating the edge probability map of a regular image and obtaining the edge probability descriptor to realize the conversion of a regular color image to a hand-drawn image. Then, a convolutional neural network is used to establish the feature library required for the hand-drawn image to perform subsequent retrieval tasks, thereby improving the accuracy of the retrieval. However, the sketch given by this method also only has a single object, and the sketch generation task cannot be expanded to the scene level. At the same time, it also has the common shortcomings of the method of using edge information to generate sketches, that is, it cannot highlight the semantic information of the object, and the generated sketch does not have diversity. Summary of the invention

[0006] In view of the shortcomings of the current scene sketch generation algorithm that cannot generate scenes that meet the characteristics of human hand-drawn sketches, and the limitations of the result sketches in reflecting the original scene image information, the purpose of the present invention is to provide a scene hand-drawn sketch generation and editing method, system, device and medium driven by semantic layout. By introducing a scene layout modeling module based on the Transformer architecture, the finally generated scene sketch meets the characteristics of loose corresponding positions of human hand-drawn sketches. By introducing a single object sketch generation module, the generation process of the diffusion model is controlled by using fusion conditions, the generation process is effectively supervised, and the quality of the model output results is improved. By introducing a sketch vectorization module, the characteristics of proper details of human hand-drawn scene sketches are effectively displayed, and the results are editable output. Finally, by connecting the various modules together, a generation model from scene images to diversified and editable hand-drawn scene sketches is formed, so that the finally generated scene sketch has the hand-drawn abstract characteristics of the scene level, and shows diversity in layout and object form.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A scene hand-drawn sketch generation and editing method based on semantic layout driving includes the following steps:

[0009] Step 1, preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image;

[0010] Step 2, constructing a scene image panorama segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module and a scene sketch combination module; wherein:

[0011] The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0012] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts;

[0013] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object;

[0014] The sketch vectorization module is used to convert the pixel-level sketch of a single object in the scene image into a vectorized sketch of the single object;

[0015] The scene sketch combination module is used to combine the vectorized sketch of a single object into multiple scene sketches according to the generated layouts of multiple scene images;

[0016] Step 3, performing model training on the scene layout modeling module and the single object sketch generation module constructed in step 2;

[0017] Step 4: construct a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module constructed in step 2, and the scene layout modeling module and single object sketch generation module trained in step 3.

[0018] The specific method of step 1 is:

[0019] Step 1.1: for multiple objects in the scene image and their corresponding sketches, Figure 1 One-to-one correspondence,consists of an image-sketch pair of a single object, where the object sketch serves as the true target image;

[0020] Step 1.2, read the image sketch pair of the single object obtained in step 1.1 through the pytorch deep learning image processing library tool, and adjust the shape of the image sketch pair of the single object to the size required by the single object sketch image generation module before inputting it into the single object sketch generation module; at the same time, normalize the image sketch pair of the single object, and map the color value range of the input image data from an integer value of [0 to 255] to a floating point value of [-1 to 1] to obtain the preprocessed image sketch image pair of the single object.

[0021] The specific method of step 2 is:

[0022] Step 2.1, the scene image panoptic segmentation module includes a panoptic segmentation network model, which is used to perform panoptic segmentation on the input scene image to obtain the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0023] Step 2.2, the scene layout modeling module includes a network model based on a transformer architecture, and the network model based on the transformer architecture includes multiple transformer layers, each transformer layer is composed of two LayerNorm normalization layers, a causal self-attention layer and an MLP multilayer perceptron; the MLP multilayer perceptron is composed of a Linear layer, a GELU activation layer and a Linear layer; by inputting a starting layout information element in the semantic layout of the original scene image obtained in step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs multiple scene image layouts predicted by the semantic layout of the original scene image;

[0024] Step 2.3, the single object sketch generation module includes an autoencoder model and a conditional diffusion generation model, the conditional diffusion generation model input is the category of all objects in the input scene image obtained in step 2.1 and the corresponding object instance segmentation map, and the single object sketch generation module output is a pixel-level single object sketch in the input scene image;

[0025] Step 2.4, the sketch vectorization module includes a differentiable rasterizer. The input of the sketch vectorization module is the pixel-level sketch of a single object in the input scene image obtained in step 2.3. For the input sketch, n Bezier curves are initialized, and the position information of the control points of these curves and the color information of the lines are used as trainable parameters. The visual effect of the final vectorized sketch composed of Bezier curves is perfectly close to the given pixel-level object sketch, so as to achieve the purpose of vectorization. The sketch vectorization module outputs a vectorized sketch of a single object.

[0026] In step 2.5, the scene sketch combination module combines multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module and the vectorized sketch of a single object output by the sketch vectorization module, and finally outputs multiple scene sketches.

[0027] The specific method of step 3 is:

[0028] Step 3.1, for the training of the scene layout modeling module, select AdamW as the parameter optimizer, set the initial learning rate, maximum number of iterations, batch number of a single iteration, and betas parameters;

[0029] The training of the single object sketch generation module is divided into two stages. In the first stage, the autoencoder model is trained, AdamW is selected as the parameter optimizer, and the initial learning rate, maximum number of iterations, batch size of a single iteration, and betas parameters are set; in the second stage, the conditional diffusion generation model is trained, and the initial learning rate, maximum number of iterations, batch size of a single iteration, and betas parameters are set;

[0030] Step 3.2, for the training of the scene layout modeling module, the following cross entropy loss function is used:

[0031]

[0032] Among them, i Indicates the value of the i-th layout element, o j Represents all element values ​​before the i-th element;

[0033] For the first stage of training in the single object sketch generation module, there are three parts of loss, namely:

[0034] L kl =D KL (z|N(z r ))

[0035] L l1 =||(inputs)-(recs)||

[0036] L perceptual =||Vgg(inputs)-Vgg(recs)||

[0037] Among them, D KL represents the KL divergence measure of latent space encoding and random Gaussian noise; L l1 It is to calculate the L1 distance between the input image and the reconstructed image, where inputs and recs represent the input image and the reconstructed image respectively; L perceptual It is the L1 distance between the features of the input image and the reconstructed image after the pre-trained model VGG, also known as the perceptual loss; the final loss function of the first stage of training is the weighted sum of each part:

[0038] L first =λ1·L l1 +λ2·L perceptual +λ3·L kl

[0039] Where λ1, λ2, λ3 are 1, 1, 10 respectively -6 ;

[0040] For the second stage training of the single object sketch generation module, the following loss function is used:

[0041] L second =E ε(x),C,f,r～N(0,1),t [||rr θ (z t ,t,C,f)|| 2 ]

[0042] Among them, C,f represents the category corresponding to a single object and the image features extracted by the feature extraction model, r represents the real noise added in the current time step, and r θ represents the Unet denoising model, r θ (z t ,t,C,f) represents the noise added to the current time step predicted by the denoising model, and the L2 distance between the two is used as the loss function of this stage;

[0043] Step 3.3, during each iteration of model training, the semantic layout of the original scene image output by step 2.1 is input into the scene layout modeling module for calculation, and the image sketch image pair of a single object output by step 1.2 is input into the single object sketch generation module for calculation, and after each calculation, the model output and the corresponding real target image output by step 1.1 are input into the loss function to calculate the size of the loss, and the model parameters are updated by back propagation; the model file is saved, and after the training is completed, the model files under different iteration numbers are tested one by one, the program is run to sample and obtain the results, and the final model with the best performance is obtained by comparison, so as to obtain the trained scene layout modeling module and single object sketch generation module.

[0044] The specific method of step 4 is:

[0045] Step 4.1, inputting the scene image into the scene image panoramic segmentation module, the panoramic segmentation network model in the scene image panoramic segmentation module loads the pre-trained model parameters, the input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0046] Step 4.2, input the semantic layout of the original scene image output by the scene image panoptic segmentation module in step 4.1 into the scene layout modeling module, define the semantic layout of the original scene image in the panoptic segmentation result of the scene image as a vector as a representation of the layout information of the scene image, and input it into the scene layout modeling module, the L transformer layer networks in the scene layout modeling module predict the value of the next layout information according to the value of the current position of the input vector and the previous value of the position, and finally obtain m new scene layouts from the original layout information, where m is a value that can be customized input;

[0047] Step 4.3, connect the scene image panoramic segmentation module and the single object sketch generation module, obtain the segmentation map of each foreground object based on the semantic layout of the original scene image output by the panoramic segmentation network model in the scene image panoramic segmentation module; the feature extraction model in the single object sketch generation module extracts the features of the object image; the conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model; the Unet denoising model fuses the random Gaussian noise, the striped noise and the image into a single object sketch generation module; the conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the random Gaussian noise, the striped noise and the image ... class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model The output of the conditional fusion module and the current time step t of the conditional diffusion generation model are used as input to predict the noise added between time step t and time step t-1, and the predicted noise is subtracted from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1. The predicted image, the output of the conditional fusion module and time step t-1 are used again using the Unet denoising model to predict the noise added between time step t-1 and time step t-2, and the predicted image of the previous time step is obtained again, and so on, until the loop iterates to the initial time step, and a sampled single object sketch is obtained;

[0048] Step 4.4, vectorize the single object sketch obtained in step 4.3. At the same time, according to the multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module, the vectorized sketch of the single object output by the sketch vectorization module is arranged on the final canvas to form the final scene sketch output; for the placement of the foreground sketch, it is placed according to the position information in the corresponding layout; and for the placement of the background sketch, pixel clustering is performed on the background segmentation map in the input scene image output in step 2.1, and the positions of K cluster centers are used as the final positions of the background sketch. The background of the same category is placed at K positions using the same generated sketch; thereby obtaining multiple scene sketches with different scene layouts.

[0049] The autoencoder model and the conditional diffusion generation model in step 2.3 specifically include:

[0050] Step 2.3.1, Autoencoder Model

[0051] The autoencoder model is a network of an encoder-decoder structure. The encoder part uses a Conv convolution layer, a Downsample layer, and a GN normalization layer to encode the image sketch image pair of a single object obtained in step 1.2 into a latent vector in the latent space; the decoder part also uses a Conv convolution layer, an Upsample layer, and a GN normalization layer to decode the latent vector in the latent space into an image in the pixel space.

[0052] Step 2.3.2, Conditional Diffusion Generation Model

[0053] The conditional diffusion generation model consists of a feature extraction model, a conditional fusion model, and a Unet denoising model;

[0054] The feature extraction model adopts the FPN feature pyramid network architecture, the bottom-to-top module in the feature extraction model adopts the Resnet50 structure, and the feature extraction model is used to extract the features of the object image;

[0055] The conditional fusion model consists of a Linear layer and a Swish activation layer, which is used to fuse the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model;

[0056] The Unet denoising model consists of down_block, mid_block and up_block. All three structures include basic Block modules, among which the down_block network structure includes the Downsample down-sampling module, and the up_block network structure includes the Upsample up-sampling module; the basic Block module consists of two Linear layers and two Conv convolutional layers, the activation layer adopts the Swish activation function, the Downsample down-sampling module consists of two convolutional modules, and the Upsample up-sampling module consists of a convolutional layer and a deconvolutional layer; in the parts of the feature maps with the same width and height in down_block and up_block, the feature maps are cascaded and fused to achieve multi-scale feature fusion.

[0057] The present invention also provides a scene hand-drawn sketch generation and editing system based on semantic layout drive, comprising:

[0058] A scene image preprocessing module, used for preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image;

[0059] The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0060] A scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts;

[0061] A single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object;

[0062] A sketch vectorization module, used to convert a pixel-level sketch of a single object in a scene image into a vectorized sketch of a single object;

[0063] A scene sketch combination module, used for combining vectorized sketches of a single object into multiple scene sketches according to the generated layouts of multiple scene images;

[0064] A model training module is used to perform model training on the scene layout modeling module and the single object sketch generation module;

[0065] The scene sketch generation model building module is used to build a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module, as well as the trained scene layout modeling module and single object sketch generation module.

[0066] The present invention also provides a scene hand-drawn sketch generation and editing device based on semantic layout drive, comprising:

[0067] Memory: a computer program storing the above-mentioned method for generating and editing a scene hand-drawn sketch driven by semantic layout, which is a computer-readable device;

[0068] Processor: used to implement the semantic layout-driven scene hand-drawn sketch generation and editing method when executing the computer program.

[0069] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for generating and editing a scene hand-drawn sketch based on semantic layout driving.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. The present invention, through the innovative design of a scene layout modeling module, a single object sketch generation module and a sketch vectorization module, is equipped with a scene image panoramic segmentation module and a scene sketch combination module to form a scene sketch generation and editing method. The method can generate a corresponding plurality of human hand-drawn scene sketches with different scene layouts, diverse object shapes and editable sketch details according to the input scene image, so that the finally generated scene sketch has the hand-drawn abstract characteristics of the scene level, and can highlight the semantic characteristics of the scene objects.

[0072] 2. The present invention introduces a scene layout modeling module based on the Transformer architecture to model the layout of the input scene image and predict multiple scene image layouts. For the same input scene image, the present invention can generate multiple human hand-drawn scene sketches with different layouts. When humans hand-draw scene sketches, they only consider the relative position relationship between objects in the scene, and humans are relatively loose in following this correspondence. The present invention can generate scene sketches with different layouts by modeling the input scene layout, which can effectively simulate the characteristics of human hand-drawn scene sketches.

[0073] 3. The present invention introduces a single object sketch generation module, which includes a feature extraction model and a conditional fusion model. The feature extraction model can effectively extract the features of the object image. The conditional fusion model fuses the features of the object image with the category information corresponding to the object, and uses its output as the input of the Unet denoising model, which can effectively supervise the generation process and improve the quality of the model output results.

[0074] 4. The present invention introduces a sketch vectorization module to perform vectorization processing on the pixel-level single object sketch in the input scene image output by the single object sketch generation module. During the vectorization process, the thickness and color of the Bezier curve can be flexibly controlled, and the features of proper detail and omission of human hand-drawn scene sketches can be effectively displayed. In addition, users can also customize these parameters to realize editing operations on the hand-drawn scene sketches.

[0075] In summary, the present invention proposes a method for generating diversified hand-drawn sketches of complex scenes driven by semantic layout. The design of the diffusion process in the latent space in the single object sketch generation module greatly shortens the sampling time of the model and improves the generation efficiency. The method not only realizes the transformation from complex scene images to sketches, but also the scene layout modeling module makes the resulting generation layout more diversified and the number of objects in the scene variable. The conditional diffusion generation module in the single object sketch generation module makes the object shape richer, and the sketch vectorization module makes the sketch abstraction level customizable. Through the above various effects, the final generated sketch is closer to human hand-drawn sketches, which makes up for the shortcomings of the existing scene sketch generation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of the method of the present invention.

[0077] Figure 2 It is a structural diagram of the scene layout modeling module of the present invention.

[0078] Figure 3 It is a structural diagram of the self-encoding model of the present invention.

[0079] Figure 4It is a structural diagram of the Unet denoising model of the present invention. DETAILED DESCRIPTION

[0080] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0081] The core content of the present invention includes the use of a transformer-based scene layout generation model, which uses its powerful natural language processing ability to predict reasonable and diversified scene layouts, so that the model can achieve a one-to-many scene sketch generation goal and realize some diversity functions of model generation; the conditional diffusion model is used to realize the sketch generation task of a single object, and the fusion features of category and local image features are used as conditions to guide the generation process, thereby avoiding the influence of the incompleteness of the local object image in the scene on the diffusion model when only the image features are used as diffusion conditions, and improving the accuracy of model generation; at the same time, the training and sampling processes of the diffusion model are both carried out in the latent space, which greatly reduces the time cost of training and sampling; the sketch vectorization model is used and applied to the key nodes in the method flow, and the sketch in the vectorized format can not only convey the complex scene information in the original image more clearly, but also avoid the limitation that the scene sketch at the pixel level is easy to blur, and the sketch in the vectorized format based on strokes is more in line with the painting characteristics of human hand-drawn sketches.

[0082] A scene hand-drawn sketch generation and editing method based on semantic layout driving includes the following steps:

[0083] Step 1, preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image;

[0084] Step 1.1, COCO dataset and SketchyCOCO dataset are two datasets used in this invention. For SketchyCOCO dataset, since sketch generation requires paired image data, for multiple objects in the scene image in SketchyCOCO dataset and the sketches corresponding to the objects, the objects and their corresponding sketches are mapped to the same image. Figure 1 One-to-one correspondence, forming an image-sketch pair of a single object, with the object sketch as the real target image; for the COCO dataset, the scene layout in each scene image in the dataset is extracted to obtain the scene image layout training set;

[0085] Step 1.2, read the image sketch pair of a single object obtained in step 1.1 through the torchvision library, and adjust the shape of the image sketch pair of a single object to 3×256×256 before inputting it into the single object sketch generation module to meet the input requirements of the single object sketch generation module; at the same time, the image sketch pair of a single object is normalized using a mean and variance of 0.5, and the color value range of the input image data is mapped from an integer value of [0 to 255] to a floating point value of [-1 to 1] to obtain an object sketch image pair training set.

[0086] Step 2, constructing a scene image panorama segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module and a scene sketch combination module; wherein:

[0087] The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map, thereby reducing the complexity of the overall task;

[0088] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts, so that the output scene sketch has layout diversity characteristics;

[0089] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object;

[0090] The sketch vectorization module is used to convert the pixel-level sketch of a single object in the scene image into a vectorized sketch of a single object, which can solve the problem of unclear sketches of small objects. At the same time, the vector map has multiple editable parameters, which can realize the user-editable generation function;

[0091] The scene sketch combination module is used to combine the vectorized sketch of a single object into multiple scene sketches according to the generated layouts of multiple scene images;

[0092] Step 2.1, the scene image panoptic segmentation module includes a panoptic segmentation network model, which is responsible for panoptic segmentation of the input scene image, thereby obtaining the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0093] The panoptic segmentation network model is mainly composed of four components: Backbone encoder backbone network, decoupled ASPP spatial pyramid pooling network, decoder network for decoupling specific tasks, and prediction head network for specific tasks. The Backbone encoder backbone network is obtained based on the ImageNet dataset pre-training, and a hole convolution is added to the last layer; the decoupled ASPP spatial pyramid pooling network is used to extract multi-scale context information; the decoder network for decoupling specific tasks introduces a 1 / 8 scale jump connection and adds a 5×5 convolution layer after upsampling; the prediction head network for specific tasks is a FCN fully convolutional network.

[0094] The present invention introduces a scene image panoramic segmentation module to perform panoramic semantic understanding of complex scene images, effectively performs semantic modeling of complex scene images, decomposes the complex scene sketch generation task into easily handleable sub-problems of single object sketch generation, and elevates sketch generation from the traditional single object level to the complex scene level, solving the complex problem of sketch generation in complex scenes.

[0095] Step 2.2, such as Figure 2 As shown, the scene layout modeling module includes a network model based on a transformer architecture, and the network model based on the transformer architecture includes multiple transformer layers, each transformer layer is composed of two LayerNorm normalization layers, a causal self-attention layer and an MLP multilayer perceptron; the MLP multilayer perceptron is composed of a Linear layer, a GELU activation layer and a Linear layer; by inputting a starting layout information element in the semantic layout of the original scene image obtained in step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs multiple scene image layouts predicted by the semantic layout of the original scene image;

[0096] Step 2.3, the single object sketch generation module includes an autoencoder model and a conditional diffusion generation model, the conditional diffusion generation model input is the category of all objects in the input scene image obtained in step 2.1 and the corresponding object instance segmentation map, and the single object sketch generation module output is a pixel-level single object sketch in the input scene image;

[0097] Step 2.3.1, Autoencoder Model

[0098] like Figure 3As shown in the figure, the autoencoder model is a network with an encoder-decoder structure. The encoder part uses a Conv convolution layer, a Downsample layer, and a GN normalization layer, which is responsible for encoding a 3×256×256 image in the pixel space into a 4×32×32 latent vector in the latent space; the decoder part also uses a Conv convolution layer, an Upsample layer, and a GN normalization layer, which is responsible for decoding the vector in the latent space into an image in the pixel space; this model allows the diffusion process to be carried out in the latent space, which can greatly reduce the time cost of training and sampling;

[0099] Step 2.3.2, Conditional Diffusion Generation Model

[0100] The conditional diffusion generation model consists of a feature extraction model, a conditional fusion model, and a Unet denoising model;

[0101] The feature extraction model adopts the FPN feature pyramid network architecture, and the bottom-to-top module in the feature extraction model adopts the Resnet50 structure. The feature extraction model is responsible for extracting the features of the object image;

[0102] The conditional fusion model consists of a Linear layer and a Swish activation layer, which is used to fuse the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model;

[0103] like Figure 4 As shown in the figure, the Unet denoising model consists of down_block, mid_block and up_block, and the three structures all include basic Block modules, among which the down_block network structure includes the Downsample down-sampling module, and the up_block network structure includes the Upsample up-sampling module; the basic Block module consists of two Linear layers and two Conv convolutional layers, and the activation layer adopts the Swish activation function to effectively solve the problems of gradient disappearance and gradient explosion in the neural network; the Downsample down-sampling module consists of two convolutional modules; the Upsample up-sampling module consists of a convolutional layer and a deconvolutional layer; in the parts of the feature maps with the same width and height in down_block and up_block, the feature maps are cascaded and fused to achieve multi-scale feature fusion;

[0104] Step 2.4, the sketch vectorization module includes a differentiable rasterizer. The input of the sketch vectorization module is a 3×256×256 pixel-level single object sketch. For the input sketch, n Bezier curves are initialized, and the position information of the control points of these curves and the color information of the lines are used as trainable parameters. The visual effect of the final vectorized sketch composed of Bezier curves is perfectly close to the given pixel-level object sketch, so as to achieve the purpose of vectorization. The sketch vectorization module outputs a vectorized sketch of a single object.

[0105] In step 2.5, the scene sketch combination module places the vectorized sketches of single objects output by the sketch vectorization module at corresponding positions according to the multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module, and finally outputs multiple scene sketches.

[0106] Step 3, performing model training on the scene layout modeling module and the single object sketch generation module constructed in step 2;

[0107] Step 3.1: In the network structure corresponding to the present invention, only the scene layout modeling module and the single object sketch generation module need to be trained.

[0108] For the training of the scene layout modeling module, AdamW is selected as the parameter optimizer, the initial learning rate is 0.000288, the maximum number of iterations is set to 25, the single iteration batch size is 64, and the betas parameter is (0.9, 0.95);

[0109] The training of the single object sketch generation module is divided into two stages. In the first stage, the autoencoder model is trained, AdamW is selected as the parameter optimizer, the initial learning rate is 0.0001, the betas parameter is (0.5, 0.9), the maximum number of iterations is set to 200, and the batch size of a single iteration is 4; in the second stage, the conditional diffusion generation model is trained, AdamW is also selected as the parameter optimizer, the initial learning rate is 0.0001, and the betas parameter is (0.5, 0.999);

[0110] Step 3.2, the loss function is used to evaluate the performance of the network model. The loss functions defined in the present invention at different training stages are as follows:

[0111] For the training of the scene modeling module, the following cross entropy loss function is used:

[0112]

[0113] Among them, i Indicates the value of the i-th layout element, o jRepresents all element values ​​before the i-th element;

[0114] For the first stage of training in the single object sketch generation module, there are three parts of loss, namely:

[0115] L kl =D KL (z|N(z r ))

[0116] L l1 =||(inputs)-(recs)||

[0117] L perceptual =||Vgg(inputs)-Vgg(recs)||

[0118] Among them, D KL represents the KL divergence measure of latent space encoding and random Gaussian noise; L l1 It is to calculate the L1 distance between the input image and the reconstructed image, where inputs and recs represent the input image and the reconstructed image respectively; L perceptual It is the L1 distance between the features of the input image and the reconstructed image after the pre-trained model VGG, also known as the perceptual loss; the final loss function of the first stage of training is the weighted sum of each part:

[0119] L first =λ1·L l1 +λ2·L perceptual +λ3·L kl

[0120] Where λ1, λ2, λ3 are 1, 1, 10 respectively -6 ;

[0121] For the second stage training of the single object sketch generation module, the following loss function is used:

[0122] L second =E ε(x),C,f,r～N(0,1),t [||rr θ (z t ,t,C,f)|| 2 ]

[0123] Among them, C,f represents the category corresponding to a single object and the image features extracted by the feature extraction model, r represents the real noise added in the current time step, and r θ represents the Unet denoising model, r θ (z t ,t,C,f) represents the noise added to the current time step predicted by the denoising model, and the L2 distance between the two is used as the loss function of this stage;

[0124] Step 3.3, for the training of the two modules in the present invention, in each iteration of model training, the scene image layout training set obtained in step 1.1 is input into the scene layout modeling module, and the object sketch image pair training set obtained in step 1.2 is input into the single object sketch generation module, and the result of the scene layout modeling module model output and the true value contained in the scene image layout training set obtained in step 1.1 are input into the loss function to calculate the loss, and the result of the single object sketch generation module output and the true value contained in the object sketch image pair training set obtained in step 1.2 are input into the loss function to calculate the loss, and the model parameters are updated by back propagation; the model file is saved every 20 iterations, and the model files under different iteration numbers are tested one by one after the training is completed, the program is run to sample and obtain the results, and the final model with the best performance is obtained by comparison, so as to obtain the trained scene layout modeling module and the single object sketch generation module.

[0125] Step 4: construct a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module constructed in step 2, and the scene layout modeling module and single object sketch generation module trained in step 3.

[0126] like Figure 1 As shown, Figure 1 The invention describes the training process of each module that needs to be trained and the complete model reasoning process.

[0127] Step 4.1, input the scene image into the scene image panoramic segmentation module, the panoramic segmentation network model in the scene image panoramic segmentation module loads the pre-trained model parameters, the input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0128] Step 4.2, input the semantic layout of the original scene image output by the scene image panoptic segmentation module in step 4.1 into the scene layout modeling module, define the semantic layout of the original scene image in the panoptic segmentation result of the scene image as a vector as a representation of the layout information of the scene image, and input it into the scene layout modeling module, the L transformer layer networks in the scene layout modeling module predict the value of the next layout information according to the value of the current position of the input vector and the previous value of the position, and finally obtain m new scene layouts from the original layout information, where m is a value that can be customized input;

[0129] Step 4.3, connect the scene image panoramic segmentation module and the single object sketch generation module, obtain the segmentation map of each foreground object based on the semantic layout of the original scene image output by the panoramic segmentation network model in the scene image panoramic segmentation module; the feature extraction model in the single object sketch generation module extracts the features of the object image; the conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model; the Unet denoising model fuses the random Gaussian noise, conditional The output of the fusion module and the current time step t of the conditional diffusion generation model are used as input to predict the noise added between time step t and time step t-1, and the predicted noise is subtracted from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1. The Unet denoising model is used again to take the predicted image, the output of the conditional fusion module and time step t-1 as input to predict the noise added between time step t-1 and time step t-2, thereby obtaining the predicted image of the previous time step again, and so on, until the loop iterates to the initial time step, and a single object sketch of the sample is obtained;

[0130] Step 4.4, vectorize the single object sketch obtained in step 4.3. At the same time, according to the multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module, the vectorized sketch of the single object output by the sketch vectorization module is arranged on the final canvas to form the final scene sketch output; for the placement of the foreground sketch, it can be placed according to the position information in the corresponding layout; and for the placement of the background sketch, pixel clustering is performed on the background segmentation map in the input scene image output in step 2.1, and the positions of K cluster centers are used as the final positions of the background sketch. The background of the same category can be placed at K positions using the same generated sketch; thereby obtaining multiple scene sketches with different scene layouts.

[0131] Model Inference

[0132] (1) Read the network model file that has been fully iteratively trained and load the model parameters into the program model. Modify the model to inference mode and fix the parameters of each network layer.

[0133] (2) Import the input scene image into the program, keeping the original image size unchanged. The program first uses the scene image panoramic segmentation module to perform panoramic segmentation on the image to obtain the initial scene image semantic layout information. The scene layout modeling module uses the initial semantic layout as input to obtain multiple new scene layouts for subsequent use; the single object sketch generation module samples a single object sketch with a shape of 3×256×256 based on the object category and object image features, and obtains a single sketch in vector format after passing through the sketch vectorization module; finally, combined with the multi-scene layout information given by the scene layout modeling module, multiple scene sketches with the same size as the input scene image are given.

[0134] (3) The two methods compared with the results obtained by the present invention are CLIPascene and Info-drawing, and the test set is from the MS-COCO dataset. The quality of the model results is judged by the LPIPS, FID and CLIP evaluation indicators, which can evaluate the similarity between images. As shown in Table 1, the arrows on the right side of each indicator indicate that the larger the value is, the higher the similarity between the two images is, and vice versa. It can be seen from Table 1 that the results obtained by the present invention perform better than the other two methods in terms of the three indicators.

[0135] Table 1 Comparison of indicators of different methods

[0136] method LPIPS(↓) FID(↓) CLIP(↑) CLIPascene 0.461 84.943 0.784 Info-drawing 0.512 73.059 0.755 Ours 0.340 24.354 0.863

[0137] The beneficial effects of the present invention are as follows: the present invention proposes a method for generating diversified hand-drawn sketches of complex scenes driven by semantic layout, and the design of the diffusion process in the latent space greatly shortens the sampling time of the model and improves the generation efficiency; the method not only realizes the transformation from complex scene images to sketches, but also can generate scene sketches with more diversified layouts, richer object shapes, variable number of objects in the scene, and customizable sketch abstraction levels, and the final generated sketch is closer to human hand-drawn sketches, which makes up for the shortcomings of existing scene sketch generation algorithms.

[0138] The present invention also provides a scene hand-drawn sketch generation and editing system based on semantic layout drive, comprising:

[0139] A scene image preprocessing module, used for preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image;

[0140] The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map;

[0141] A scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts;

[0142] A single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object;

[0143] A sketch vectorization module, used to convert a pixel-level sketch of a single object in a scene image into a vectorized sketch of a single object;

[0144] A scene sketch combination module, used for combining vectorized sketches of a single object into multiple scene sketches according to the generated layouts of multiple scene images;

[0145] A model training module is used to perform model training on the scene layout modeling module and the single object sketch generation module;

[0146] The scene sketch generation model building module is used to build a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module, as well as the trained scene layout modeling module and single object sketch generation module.

[0147] The present invention also provides a scene hand-drawn sketch generation and editing device based on semantic layout drive, comprising:

[0148] Memory: a computer program storing the above-mentioned method for generating and editing a scene hand-drawn sketch driven by semantic layout, which is a computer-readable device;

[0149] Processor: used to implement the semantic layout-driven scene hand-drawn sketch generation and editing method when executing the computer program.

[0150] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for generating and editing a scene hand-drawn sketch based on semantic layout driving.

Claims

1. A scene hand-drawn sketch generation and editing method based on semantic layout drive, characterized in that: The following steps are involved: Step 1, preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image; Step 2, constructing a scene image panorama segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module and a scene sketch combination module; wherein: The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map; The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts; The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object; The sketch vectorization module is used to convert the pixel-level sketch of a single object in the scene image into a vectorized sketch of the single object; The scene sketch combination module is used to combine the vectorized sketch of a single object into multiple scene sketches according to the generated layouts of multiple scene images; Step 3, performing model training on the scene layout modeling module and the single object sketch generation module constructed in step 2; Step 4: construct a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module constructed in step 2, and the scene layout modeling module and single object sketch generation module trained in step 3.

2. The method for generating and editing scene hand-drawn sketches based on semantic layout drive according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1, for multiple objects in the scene image and the sketches corresponding to the objects, the objects and their corresponding sketches are matched one by one to form an image-sketch pair of a single object, and the object sketch is used as the real target image; Step 1.2, read the image sketch pair of the single object obtained in step 1.1 through the pytorch deep learning image processing library tool, and adjust the shape of the image sketch pair of the single object to the size required by the single object sketch generation module before inputting it into the single object sketch generation module; at the same time, normalize the image sketch pair of the single object, and map the color value range of the input image data from an integer value of [0 to 255] to a floating point value of [-1 to 1] to obtain the preprocessed image sketch image pair of the single object.

3. The method for generating and editing scene hand-drawn sketches based on semantic layout drive according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1, the scene image panoptic segmentation module includes a panoptic segmentation network model, which is used to perform panoptic segmentation on the input scene image to obtain the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map; Step 2.2, the scene layout modeling module includes a network model based on a transformer architecture, and the network model based on the transformer architecture includes multiple transformer layers, each transformer layer consists of two LayerNorm normalization layers, a causal self-attention layer and an MLP multilayer perceptron; the MLP multilayer perceptron consists of a Linear layer, a GELU activation layer and a Linear layer; by inputting a starting layout information element in the semantic layout of the original scene image obtained in step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs multiple scene image layouts predicted by the semantic layout of the original scene image; Step 2.3, the single object sketch generation module includes an autoencoder model and a conditional diffusion generation model, the conditional diffusion generation model input is the category of all objects in the input scene image obtained in step 2.1 and the corresponding object instance segmentation map, and the single object sketch generation module output is a pixel-level single object sketch in the input scene image; Step 2.4, the sketch vectorization module includes a differentiable rasterizer. The input of the sketch vectorization module is the pixel-level sketch of a single object in the input scene image obtained in step 2.

3. For the input sketch, n Bezier curves are initialized, and the position information of the control points of these curves and the color information of the lines are used as trainable parameters. The visual effect of the final vectorized sketch composed of Bezier curves is perfectly close to the given pixel-level object sketch, so as to achieve the purpose of vectorization. The sketch vectorization module outputs a vectorized sketch of a single object. In step 2.5, the scene sketch combination module combines multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module and the vectorized sketch of a single object output by the sketch vectorization module, and finally outputs multiple scene sketches.

4. The method for generating and editing scene hand-drawn sketches based on semantic layout drive according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1, for the training of the scene layout modeling module, select AdamW as the parameter optimizer, set the initial learning rate, maximum number of iterations, batch number of a single iteration, and betas parameters; The training of the single object sketch generation module is divided into two stages. In the first stage, the autoencoder model is trained, AdamW is selected as the parameter optimizer, and the initial learning rate, maximum number of iterations, batch size of a single iteration, and betas parameters are set; in the second stage, the conditional diffusion generation model is trained, and the initial learning rate, maximum number of iterations, batch size of a single iteration, and betas parameters are set; Step 3.2, for the training of the scene layout modeling module, the following cross entropy loss function is used: Among them, i Indicates the value of the i-th layout element, o j Represents all element values ​​before the i-th element; For the first stage of training in the single object sketch generation module, there are three parts of loss, namely: L kl =D KL (z|N(z r )) L l1 =||(inputs)-(recs)|| L perceptual =||Vgg(inputs)-Vgg(recs)|| Among them, D KL represents the KL divergence measure of latent space encoding and random Gaussian noise; L l1 It is to calculate the L1 distance between the input image and the reconstructed image, where inputs and recs represent the input image and the reconstructed image respectively; L perceptual It is the L1 distance between the features of the input image and the reconstructed image after the pre-trained model VGG, also known as the perceptual loss; the final loss function of the first stage of training is the weighted sum of each part: L first =λ1·L l1 +λ2·L perceptual +λ3·L kl Where λ1, λ2, λ3 are 1, 1, 10 respectively -6 ; For the second stage training of the single object sketch generation module, the following loss function is used: L second =E ε(x),C,f,r～N(0,1),t [||r-r θ (z t ,t,C,f)|| 2 ] Among them, C,f represents the category corresponding to a single object and the image features extracted by the feature extraction model, r represents the real noise added in the current time step, and r θ represents the Unet denoising model, r θ (z t ,t,C,f) represents the noise added to the current time step predicted by the denoising model, and the L2 distance between the two is used as the loss function of this stage; Step 3.3, during each iteration of model training, the semantic layout of the original scene image output by step 2.1 is input into the scene layout modeling module for calculation, and the image sketch image pair of a single object output by step 1.2 is input into the single object sketch generation module for calculation, and after each calculation, the model output and the corresponding real target image output by step 1.1 are input into the loss function to calculate the size of the loss, and the model parameters are updated by back propagation; the model file is saved, and after the training is completed, the model files under different iteration numbers are tested one by one, the program is run to sample and obtain the results, and the final model with the best performance is obtained by comparison, so as to obtain the trained scene layout modeling module and single object sketch generation module.

5. The method for generating and editing scene hand-drawn sketches based on semantic layout drive according to claim 1, characterized in that: The specific method of step 4 is: Step 4.1, inputting the scene image into the scene image panoramic segmentation module, the panoramic segmentation network model in the scene image panoramic segmentation module loads the pre-trained model parameters, the input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map; Step 4.2, input the semantic layout of the original scene image output by the scene image panoramic segmentation module in step 4.1 into the scene layout modeling module, define the semantic layout of the original scene image in the panoramic segmentation result of the scene image as a vector as a representation of the layout information of the scene image, and input it into the scene layout modeling module, the L transformer layer networks in the scene layout modeling module predict the value of the next layout information according to the value of the current position of the input vector and the previous value of the position, and finally obtain m new scene layouts from the original layout information, where m is a value that can be customized input; Step 4.3, connect the scene image panoramic segmentation module and the single object sketch generation module, obtain the segmentation map of each foreground object based on the semantic layout of the original scene image output by the panoramic segmentation network model in the scene image panoramic segmentation module; the feature extraction model in the single object sketch generation module extracts the features of the object image; the conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model; the Unet denoising model fuses the random Gaussian noise, the striped noise and the image into a single object sketch generation module; the conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the random Gaussian noise, the striped noise and the image ... class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single object sketch generation module; the conditional fusion model fuses the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model The output of the conditional fusion module and the current time step t of the conditional diffusion generation model are used as input to predict the noise added between time step t and time step t-1, and the predicted noise is subtracted from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1. The predicted image, the output of the conditional fusion module and time step t-1 are used again using the Unet denoising model to predict the noise added between time step t-1 and time step t-2, and the predicted image of the previous time step is obtained again, and so on, until the loop iterates to the initial time step, and a sampled single object sketch is obtained; Step 4.4, vectorize the single object sketch obtained in step 4.

3. At the same time, according to the multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module, the vectorized sketch of the single object output by the sketch vectorization module is arranged on the final canvas to form the final scene sketch output; for the placement of the foreground sketch, it is placed according to the position information in the corresponding layout; and for the placement of the background sketch, pixel clustering is performed on the background segmentation map in the input scene image output in step 2.1, and the positions of K cluster centers are used as the final positions of the background sketch. The background of the same category is placed at K positions using the same generated sketch; thereby obtaining multiple scene sketches with different scene layouts.

6. The method for generating and editing scene hand-drawn sketches based on semantic layout drive according to claim 3, characterized in that: The autoencoder model and the conditional diffusion generation model in step 2.3 specifically include: Step 2.3.1, Autoencoder Model The autoencoder model is a network of an encoder-decoder structure. The encoder part uses a Conv convolution layer, a Downsample layer, and a GN normalization layer to encode the image sketch image pair of a single object obtained in step 1.2 into a latent vector in the latent space; the decoder part also uses a Conv convolution layer, an Upsample layer, and a GN normalization layer to decode the latent vector in the latent space into an image in the pixel space. Step 2.3.2, Conditional Diffusion Generation Model The conditional diffusion generation model consists of a feature extraction model, a conditional fusion model, and a Unet denoising model; The feature extraction model adopts the FPN feature pyramid network architecture, the bottom-to-top module in the feature extraction model adopts the Resnet50 structure, and the feature extraction model is used to extract the features of the object image; The conditional fusion model consists of a Linear layer and a Swish activation layer, which is used to fuse the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into one piece as the conditional input of the conditional diffusion generation model; The Unet denoising model consists of down_block, mid_block and up_block. All three structures include basic Block modules, among which the down_block network structure includes the Downsample down-sampling module, and the up_block network structure includes the Upsample up-sampling module; the basic Block module consists of two Linear layers and two Conv convolutional layers, the activation layer adopts the Swish activation function, the Downsample down-sampling module consists of two convolutional modules, and the Upsample up-sampling module consists of a convolutional layer and a deconvolutional layer; in the parts of the feature maps with the same width and height in down_block and up_block, the feature maps are cascaded and fused to achieve multi-scale feature fusion.

7. A semantic layout driven scene hand-drawn sketch generation and editing system based on the method according to any one of claims 1 to 6, characterized in that: include: A scene image preprocessing module, used for preprocessing the scene image to obtain an image sketch image pair of a single object in the preprocessed scene image; The scene image panoptic segmentation module is used to perform panoptic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map; A scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts; A single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize object sketches of various forms corresponding to the same object; A sketch vectorization module, used to convert a pixel-level sketch of a single object in a scene image into a vectorized sketch of a single object; A scene sketch combination module, used for combining vectorized sketches of a single object into multiple scene sketches according to the generated layouts of multiple scene images; A model training module is used to perform model training on the scene layout modeling module and the single object sketch generation module; The scene sketch generation model building module is used to build a complete scene sketch generation model based on the scene image panoramic segmentation module, sketch vectorization module and scene sketch combination module, as well as the trained scene layout modeling module and single object sketch generation module.

8. A scene hand-drawn sketch generation and editing device based on semantic layout drive, characterized in that: include: Memory: a computer program storing the method for generating and editing a scene hand-drawn sketch based on semantic layout drive according to any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement the semantic layout-driven scene hand-drawn sketch generation and editing method described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the method for generating and editing a scene hand-drawn sketch based on semantic layout drive as described in any one of claims 1 to 6.

Images