A method and system for generating and editing scene hand-drawn sketches based on semantic layout

Abstract

A method, system, device, and medium for generating and editing scene hand-drawn sketches based on semantic layout are disclosed, comprising: scene image preprocessing to obtain image sketch pairs of individual objects in the preprocessed scene image; constructing a scene image panoramic segmentation module, a scene layout modeling module, an individual object sketch generation module, a sketch vectorization module, and a scene sketch combination module; training the scene layout modeling module and the individual object sketch generation module; and constructing a complete scene sketch generation model. The system, device, and medium are used to implement the method. This invention can generate multiple corresponding human hand-drawn scene sketches with different scene layouts, diverse object shapes, and editable sketch details based on the input scene image, so that the final generated scene sketches have scene-level hand-drawn abstract characteristics while highlighting the semantic features of scene objects.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and computer graphics, specifically relating to a method, system, device, and medium for generating and editing scene hand-drawn sketches based on semantic layout. Background Technology

[0002] Scene sketching is an important research direction in computer vision. Its applications are wide-ranging. For example, hand-drawn scene sketches in 3D modeling help architects analyze, design, and present their work; in the gaming industry, game scene designers can quickly build beautiful game scenes based on pre-defined sketches, significantly reducing development costs and improving efficiency. Therefore, generating hand-drawn scene sketches is a very promising research topic. However, scene sketching is also a very challenging task. A scene may contain various foreground and background relationships, such as a single object against a simple background, a single object against a complex background, or multiple objects against complex backgrounds. Therefore, it is very difficult for a computer to understand these complex scene relationships and generate a reasonable sketch.

[0003] Existing work on sketching primarily relies on deep learning methods, utilizing convolutional neural networks to extract image features from both the original image and the sketch. Compared to traditional sketch generation algorithms, these methods can generate higher-quality sketches. However, existing methods also have some limitations. For example, sketch generation only focuses on individual objects and does not extend the generation task to the scene level; technically, most existing methods use edge detection to extract edge information from the original image, resulting in generated sketches that visually convey more of the object's geometric features and fail to highlight semantic features.

[0004] Patent application CN106126581B discloses a deep learning-based method for retrieving hand-drawn sketch images. This invention uses edge contour detection and non-maximum suppression techniques to convert color images into sketch-like images. Then, deep learning is used to learn the discriminative depth features of the query sketch and the sketch-like image. Finally, the learned depth features are used for sketch-based image retrieval. However, the converted images only contain a single foreground object, resulting in sketch-like images that only generate sketches of a single object, failing to generate sketches at the scene level. Furthermore, because edge detection is used, the generated sketch-like images only reflect geometric edge features, neglecting semantic features, and the generated sketches lack the characteristics of human hand-drawing.

[0005] Patent application CN108009286A discloses a deep learning-based sketch retrieval method. This invention uses a method of calculating edge probability maps of regular images and obtaining edge probability descriptors to convert color regular images into hand-drawn-like images. Then, a convolutional neural network is used to build a feature library required for hand-drawn images to perform subsequent retrieval tasks, improving retrieval accuracy. However, the sketches generated by this method still only contain single objects and cannot be extended to the scene level. It also suffers from the common drawbacks of methods that use edge information to generate sketches: they fail to highlight the semantic information of objects, and the generated sketches lack diversity. Summary of the Invention

[0006] To address the shortcomings of current scene sketch generation algorithms in generating sketches that fail to conform to the characteristics of human hand-drawn sketches and the limitations in reflecting the original scene image information, this invention aims to provide a method, system, device, and medium for generating and editing scene hand-drawn sketches based on semantic layout. By introducing a scene layout modeling module based on the Transformer architecture, the generated scene sketches conform to the loosely spaced characteristics of human hand-drawn sketches. By introducing a single object sketch generation module and using fusion conditions to control the generation process of the diffusion model, the generation process is effectively supervised, improving the quality of the model output. By introducing a sketch vectorization module, the appropriate level of detail characteristic of human hand-drawn scene sketches is effectively displayed, and the results are made editable. Finally, by connecting all modules together, a generative model is formed from scene images to diverse and editable hand-drawn scene sketches, enabling the final generated scene sketches to possess scene-level hand-drawn abstract characteristics while exhibiting diversity in layout and object form.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for generating and editing scene hand-drawn sketches based on semantic layout includes the following steps:

[0009] Step 1: Scene image preprocessing, to obtain image sketch pairs of individual objects in the preprocessed scene image;

[0010] Step 2: Construct a scene image panoramic segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module, and a scene sketch combination module; wherein:

[0011] The scene image panoramic segmentation module is used to perform panoramic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0012] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts.

[0013] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize the same object corresponding to various forms of object sketches.

[0014] The Sketch Vectorization module is used to transform pixel-level sketches of individual objects in a scene image into vectorized sketches of those objects.

[0015] The Scene Sketch Combination module is used to combine vectorized sketches of a single object into multiple scene sketches based on the layout of multiple generated scene images;

[0016] Step 3: Train the models for the scene layout modeling module and the single object sketch generation module constructed in Step 2;

[0017] Step 4: Based on the scene image panoramic segmentation module, sketch vectorization module, and scene sketch combination module built in Step 2, as well as the scene layout modeling module and single object sketch generation module trained in Step 3, construct a complete scene sketch generation model.

[0018] The specific method for step 1 is as follows:

[0019] Step 1.1: For multiple objects in the scene image and their corresponding sketches, combine the objects and their corresponding sketches... Figure 1 One-to-one correspondence constitutes an image sketch pair for a single object, with the object sketch serving as the real target image;

[0020] Step 1.2: The image sketch pair of a single object obtained in Step 1.1 is read using the PyTorch deep learning image processing library tool. Before inputting it into the single object sketch generation module, the shape of the image sketch pair of the single object is adjusted to the size required by the single object sketch image generation module. At the same time, the image sketch pair of the single object is normalized. The color value range of the input image data is mapped from an integer value of [0~255] to a floating-point value of [-1~1] to obtain the preprocessed image sketch pair of the single object.

[0021] The specific method for step 2 is as follows:

[0022] Step 2.1, the scene image panoramic segmentation module includes a panoramic segmentation network model, which is used to perform panoramic segmentation on the input scene image to obtain the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0023] Step 2.2, the scene layout modeling module includes a network model based on the transformer architecture. The network model based on the transformer architecture includes multiple transformer layers. Each transformer layer consists of two LayerNorm normalization layers, a causal self-attention layer, and an MLP multilayer perceptron. The MLP multilayer perceptron consists of a linear layer, a GELU activation layer, and a linear layer. By inputting a starting layout information element from the semantic layout of the original scene image obtained in Step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs multiple scene image layouts predicted by the semantic layout of the original scene image.

[0024] Step 2.3, the single object sketch generation module includes an autoencoder model and a conditional diffusion generation model. The input of the conditional diffusion generation model is the category of all objects in the input scene image obtained in step 2.1 and the corresponding object instance segmentation map. The output of the single object sketch generation module is a pixel-level single object sketch in the input scene image.

[0025] Step 2.4: The sketch vectorization module includes a differentiable rasterizer. The input to the sketch vectorization module is a pixel-level sketch of a single object in the input scene image obtained in Step 2.3. For the input sketch, n Bézier curves are initialized. The position information of the control points of these curves and the color information of the lines are used as trainable parameters. The module continuously optimizes and iterates so that the visual effect of the final vectorized sketch composed of Bézier curves perfectly approximates the given pixel-level object sketch, thus achieving the purpose of vectorization. The output of the sketch vectorization module is a vectorized sketch of a single object.

[0026] Step 2.5: The scene sketch combination module combines multiple scene image layouts predicted by the semantic layout of the original scene image from the scene layout modeling module and the vectorized sketches of a single object output by the sketch vectorization module, and finally outputs multiple scene sketches.

[0027] The specific method for step 3 is as follows:

[0028] Step 3.1: For training the scene layout modeling module, AdamW is selected as the parameter optimizer, and the initial learning rate, maximum number of iterations, batch size per iteration, and betas parameter are set.

[0029] The training of the single object sketch generation module is divided into two stages. The first stage trains the autoencoder model, selecting AdamW as the parameter optimizer and setting the initial learning rate, maximum number of iterations, batch size per iteration, and betas parameter. The second stage trains the conditional diffusion generation model, setting the initial learning rate, maximum number of iterations, batch size per iteration, and betas parameter.

[0030] Step 3.2, for training the scene layout modeling module, the following cross-entropy loss function is used:

[0031]

[0032] Among them, o i This represents the value of the i-th layout element, o j This represents the values ​​of all elements preceding the i-th element;

[0033] For the first stage of training in the single-object sketch generation module, there are three loss components:

[0034] L kl =D KL (z|N(z r ))

[0035] L l1 =||(inputs)-(recs)||

[0036] L perceptual =||Vgg(inputs)-Vgg(recs)||

[0037] Among them, D KL The KL divergence metric represents latent space coding and random Gaussian noise; L l1 It calculates the L1 distance between the input image and the reconstructed image, where inputs and recs represent the input image and the reconstructed image, respectively; L perceptual It calculates the L1 distance between the features of the input image and the reconstructed image after processing by the pre-trained VGG model, also known as perceptual loss; the final loss function of the first stage of training is a weighted sum of the parts:

[0038] L first =λ1·L l1 +λ2·L perceptual +λ3·L kl

[0039] Where λ1, λ2, and λ3 are taken as 1, 1, and 10, respectively. -6 ;

[0040] For the second stage of training of the single-object sketch generation module, the following loss function is used:

[0041] L second =E ε(x),C,f,r～N(0,1),t [||rr θ (z t ,t,C,f)|| 2 ]

[0042] Where C,f represents the category corresponding to a single object and the image features extracted by the feature extraction model, and r represents the real noise added in the current time step. θ Represents the Unet denoising model, r θ (z t ,t,C,f) represents the noise added at the current time step predicted by the denoising model, and the L2 distance between the two is used as the loss function for this stage;

[0043] Step 3.3: In each iteration of model training, the semantic layout of the original scene image output in Step 2.1 is input into the scene layout modeling module for calculation, and the image sketches of individual objects output in Step 1.2 are input into the individual object sketch generation module for calculation. After each calculation, the model output and the corresponding real target image output in Step 1.1 are input into the loss function to calculate the magnitude of the loss, and the model parameters are updated by backpropagation. The model file is saved, and after training, the model files under different iteration numbers are tested one by one. The program is run to sample and obtain the results. The results are compared to obtain the model with the best performance. The trained scene layout modeling module and individual object sketch generation module are obtained.

[0044] The specific method for step 4 is as follows:

[0045] Step 4.1: Input the scene image into the scene image panoramic segmentation module. The panoramic segmentation network model in the scene image panoramic segmentation module loads the pre-trained model parameters. The input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0046] Step 4.2: Input the semantic layout of the original scene image output by the scene image panoramic segmentation module in Step 4.1 into the scene layout modeling module. Define the semantic layout of the original scene image in the panoramic segmentation result of the scene image as a vector as a representation of the layout information of the scene image, and input it into the scene layout modeling module. The L transformer layer networks in the scene layout modeling module predict the value of the layout information of the next position based on the value of the current position of the input vector and the value before that position. Finally, m new scene layouts are obtained from the original layout information, where m is a customizable input value.

[0047] Step 4.3: Connect the scene image panoramic segmentation module and the single object sketch generation module. Based on the semantic layout of the original scene image output by the panoramic segmentation network model in the scene image panoramic segmentation module, obtain the segmentation map of each foreground object. The feature extraction model in the single object sketch generation module extracts the features of the object image. The conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in Step 2.1 and the features of the object image extracted by the feature extraction model into a single piece as the conditional input of the conditional diffusion generation model. The Unet denoising model removes random Gaussian noise, barcodes, and other noises. The output of the object fusion module and the current time step t of the conditional diffusion generation model are used as input to predict the noise added between time step t and time step t-1. The predicted noise is subtracted from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1. The Unet denoising model is used again to predict the noise added between time step t-1 and time step t-2, and the predicted image of the previous time step is obtained again. This process is repeated until the initial time step is reached, and a sampled sketch of a single object is obtained.

[0048] Step 4.4: The single object sketch obtained in Step 4.3 is vectorized. Simultaneously, based on the multiple scene image layouts predicted from the semantic layout of the original scene image by the scene layout modeling module, the vectorized sketches of the single objects output by the sketch vectorization module are organized onto the final canvas to form the final scene sketch output. For the placement of the foreground sketch, it is placed according to the position information in the corresponding layout. For the placement of the background sketch, pixel clustering is performed on the background segmentation map of the input scene image output in Step 2.1, and the positions of K cluster centers are used as the final positions of the background sketch. Backgrounds of the same category are placed at the K positions using the same generated sketch. Thus, multiple scene sketches with different scene layouts are obtained.

[0049] The autoencoder model and the conditional diffusion generation model in step 2.3 specifically include:

[0050] Step 2.3.1, Autoencoder Model

[0051] The autoencoder model is an encoder-decoder network. The encoder part uses Conv convolutional layers, downsampled layers, and GN normalization layers to encode the preprocessed image sketches of a single object obtained in step 1.2 into latent vectors in the latent space. The decoder part also uses Conv convolutional layers, upsampled layers, and GN normalization layers to decode the latent vectors in the latent space into images in the pixel space.

[0052] Step 2.3.2, Conditional Diffusion Generation Model

[0053] The conditional diffusion generation model consists of a feature extraction model, a conditional fusion model, and a Unet denoising model.

[0054] The feature extraction model adopts the FPN feature pyramid network architecture, and the bottom-up modules in the feature extraction model adopt the ResNet50 structure. The feature extraction model is used to extract features from the body image.

[0055] The conditional fusion model consists of a linear layer and a Swish activation layer, which is used to fuse the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single conditional input for the conditional diffusion generation model.

[0056] The Unet denoising model consists of down_block, mid_block, and up_block, each including a basic Block module. The down_block network structure includes a downsample module, and the up_block network structure includes an upsample module. Each basic Block module consists of two linear layers and two convolutional layers, with the activation function being Swish. The downsample module consists of two convolutional modules, and the upsample module consists of one convolutional layer and one deconvolutional layer. In the down_block and up_block, feature maps with the same width and height are concatenated and fused, achieving multi-scale feature fusion.

[0057] This invention also provides a scene hand-drawn sketch generation and editing system based on semantic layout, including:

[0058] The scene image preprocessing module is used to preprocess the scene image to obtain image sketch pairs of individual objects in the preprocessed scene image.

[0059] The scene image panoramic segmentation module is used to perform panoramic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0060] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts.

[0061] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize the same object corresponding to various forms of object sketches.

[0062] The Sketch Vectorization module is used to convert pixel-level sketches of individual objects in a scene image into vectorized sketches of individual objects.

[0063] The Scene Sketch Combination Module is used to combine vectorized sketches of a single object into multiple scene sketches based on the layout of multiple generated scene images;

[0064] The model training module is used to train the models for the scene layout modeling module and the single object sketch generation module.

[0065] The scene sketch generation model building module is used to implement a scene image panoramic segmentation module, a sketch vectorization module, and a scene sketch combination module, as well as a trained scene layout modeling module and a single object sketch generation module, to build a complete scene sketch generation model.

[0066] The present invention also provides a scene hand-drawn sketch generation and editing device based on semantic layout, comprising:

[0067] Memory: A computer program that stores the above-mentioned method for generating and editing scene hand-drawn sketches based on semantic layout, and is a computer-readable device;

[0068] Processor: Used to implement the semantic layout-based scene hand-drawn sketch generation and editing method when executing the computer program.

[0069] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the aforementioned method for generating and editing scene hand-drawn sketches based on semantic layout.

[0070] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0071] 1. This invention, through the innovative design of a scene layout modeling module, a single object sketch generation module, and a sketch vectorization module, combined with a scene image panoramic segmentation module and a scene sketch combination module, forms a method for scene sketch generation and editing. This method can generate multiple corresponding human-hand-drawn scene sketches with different scene layouts, diverse object shapes, and editable sketch details based on the input scene image. This results in the final generated scene sketches possessing scene-level hand-drawn abstract characteristics while highlighting the semantic features of scene objects.

[0072] 2. This invention introduces a scene layout modeling module based on the Transformer architecture to model the layout of the input scene image and predict multiple scene image layouts. For the same input scene image, this invention can generate multiple human-drawn scene sketches with different layouts. When humans draw scene sketches, they only consider the relative positional relationships between objects in the scene, and their adherence to these relationships is relatively loose. This invention, by modeling the input scene layout, can generate scene sketches with different layouts, effectively simulating the characteristics of human-drawn scene sketches.

[0073] 3. This invention introduces a single object sketch generation module, which includes a feature extraction model and a conditional fusion model. The feature extraction model can effectively extract the features of the object image, and the conditional fusion model fuses the features of the object image and the corresponding category information of the object, and uses its output as the input of the Unet denoising model, which can effectively supervise the generation process and improve the quality of the model output results.

[0074] 4. This invention introduces a sketch vectorization module to vectorize the pixel-level sketches of individual objects in the input scene image output by the single object sketch generation module. During the vectorization process, the thickness and color of the Bézier curves can be flexibly controlled, effectively demonstrating the appropriate level of detail in human hand-drawn scene sketches. In addition, users can customize these parameters to achieve editing operations on the hand-drawn scene sketches.

[0075] In summary, this invention proposes a method for generating diverse hand-drawn sketches of complex scenes based on semantic layout. The design of a diffusion process in the latent space within the single object sketch generation module significantly shortens the model's sampling time and improves generation efficiency. This method not only realizes the transformation from complex scene images to sketches, but also allows for more diverse generated layouts and variable numbers of objects in the scene through the scene layout modeling module. The conditional diffusion generation module within the single object sketch generation module enriches the object shapes, and the sketch vectorization module allows for customizable sketch abstraction levels. Through these various effects, the final generated sketches are closer to human hand-drawn sketches, overcoming the shortcomings of existing scene sketch generation algorithms. Attached Figure Description

[0076] Figure 1 This is a flowchart of the method of the present invention.

[0077] Figure 2 This is a structural diagram of the scene layout modeling module of the present invention.

[0078] Figure 3 This is a structural diagram of the self-encoding model of the present invention.

[0079] Figure 4This is a structural diagram of the Unet denoising model of the present invention. Detailed Implementation

[0080] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0081] The core content of this invention includes the use of a transformer-based scene layout generation model, which leverages its powerful natural language processing capabilities to predict reasonable and diverse scene layouts, enabling the model to achieve one-to-many scene sketch generation and realizing some diversity functions in model generation; the use of a conditional diffusion model to achieve single-object sketch generation, using the fusion features of category and local image features as conditions to guide the generation process, avoiding the impact of incomplete local object images in the scene on the diffusion model when only image features are used as diffusion conditions, thus improving the accuracy of model generation; at the same time, the training and sampling processes of the diffusion model are carried out in the latent space, which greatly reduces the time cost of training and sampling; and the use of a sketch vectorization model, which is applied to key nodes in the method flow. The vectorized sketch format can more clearly convey the complex scene information in the original image, while avoiding the limitation of pixel-level scene sketches being easily blurred, and the stroke-based vectorized sketch format is more in line with the drawing characteristics of human hand-drawn sketches.

[0082] A method for generating and editing scene hand-drawn sketches based on semantic layout includes the following steps:

[0083] Step 1: Scene image preprocessing, to obtain image sketch pairs of individual objects in the preprocessed scene image;

[0084] Step 1.1: The COCO dataset and the SketchyCOCO dataset are the two datasets used in this invention. For the SketchyCOCO dataset, since sketch generation requires paired image data, for multiple objects in the scene images within the SketchyCOCO dataset, and their corresponding sketches, the objects and their corresponding sketches are... Figure 1 One-to-one correspondence constitutes an image sketch pair for a single object, with the object sketch serving as the real target image; for the COCO dataset, the scene layouts in each scene image in the dataset are extracted to obtain the scene image layout training set;

[0085] Step 1.2: The image sketch pairs of a single object obtained in Step 1.1 are read using the torchvision library. Before inputting them into the single object sketch generation module, the shape of the image sketch pairs of a single object is adjusted to 3×256×256 to meet the input requirements of the single object sketch generation module. At the same time, the image sketch pairs of a single object are normalized using a mean and variance of 0.5. The color value range of the input image data is mapped from an integer value of [0~255] to a floating-point value of [-1~1] to obtain the training set of object sketch image pairs.

[0086] Step 2: Construct a scene image panoramic segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module, and a scene sketch combination module; wherein:

[0087] The scene image panoramic segmentation module is used to perform panoramic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map, thereby reducing the overall task complexity.

[0088] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts, so that the output scene sketch has layout diversity characteristics.

[0089] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize the same object corresponding to various forms of object sketches.

[0090] The sketch vectorization module is used to transform pixel-level sketches of individual objects in a scene image into vectorized sketches of individual objects. This can solve the problem of blurry sketches of small objects. At the same time, the vector graphics have multiple editable parameters, enabling users to edit and generate them.

[0091] The Scene Sketch Combination module is used to combine vectorized sketches of a single object into multiple scene sketches based on the layout of multiple generated scene images;

[0092] Step 2.1, the scene image panoramic segmentation module includes a panoramic segmentation network model. The panoramic segmentation network model is responsible for performing panoramic segmentation on the input scene image, thereby obtaining the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0093] The panoramic segmentation network model mainly consists of four components: a backbone encoder network, a decoupled ASPP spatial pyramid pooling network, a decoder network for task-specific decoupling, and a prediction head network for task-specific purposes. The backbone encoder network is pre-trained on the ImageNet dataset, with dilated convolutions added to the last layer. The decoupled ASPP spatial pyramid pooling network is used to extract multi-scale contextual information. The decoder network for task-specific decoupling introduces 1 / 8-scale skip connections and adds 5×5 convolutional layers after upsampling. The prediction head network for task-specific purposes is an FCN fully convolutional network.

[0094] This invention introduces a panoramic scene image segmentation module to perform panoramic semantic understanding on complex scene images, effectively performing semantic modeling of complex scene images, decomposing the complex scene sketch generation task into easily manageable sub-problems of single object sketch generation, and elevating sketch generation from the traditional single object level to the complex scene level, thus solving the complex problem of sketch generation in complex scenes.

[0095] Step 2.2, as follows Figure 2 As shown, the scene layout modeling module includes a network model based on the transformer architecture. The network model based on the transformer architecture includes multiple transformer layers. Each transformer layer consists of two LayerNorm normalization layers, a causal self-attention layer, and an MLP multilayer perceptron. The MLP multilayer perceptron consists of a linear layer, a GELU activation layer, and a linear layer. By inputting a starting layout information element from the semantic layout of the original scene image obtained in step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs multiple scene image layouts predicted by the semantic layout of the original scene image.

[0096] Step 2.3, the single object sketch generation module includes an autoencoder model and a conditional diffusion generation model. The input of the conditional diffusion generation model is the category of all objects in the input scene image obtained in step 2.1 and the corresponding object instance segmentation map. The output of the single object sketch generation module is a pixel-level single object sketch in the input scene image.

[0097] Step 2.3.1, Autoencoder Model

[0098] like Figure 3As shown, the autoencoder model is an encoder-decoder network. The encoder part uses Conv convolutional layers, downsampled layers, and GN normalization layers to encode a 3×256×256 image in the pixel space into a 4×32×32 latent vector in the latent space. The decoder part also uses Conv convolutional layers, upsampled layers, and GN normalization layers to decode the vectors in the latent space into an image in the pixel space. This model allows the diffusion process to take place in the latent space, which can significantly reduce the time cost of training and sampling.

[0099] Step 2.3.2, Conditional Diffusion Generation Model

[0100] The conditional diffusion generation model consists of a feature extraction model, a conditional fusion model, and a Unet denoising model.

[0101] The feature extraction model adopts the FPN feature pyramid network architecture, and the bottom-up module in the feature extraction model adopts the ResNet50 structure. This feature extraction model is responsible for extracting features from the body image.

[0102] The conditional fusion model consists of a linear layer and a Swish activation layer, which is used to fuse the category information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a single conditional input for the conditional diffusion generation model.

[0103] like Figure 4 As shown, the Unet denoising model consists of down_block, mid_block, and up_block. Each of these three structures includes a basic Block module. The down_block network structure includes a downsample module, and the up_block network structure includes an upsample module. The basic Block module consists of two linear layers and two convolutional layers, with the activation layer using the Swish activation function to effectively address the vanishing and exploding gradient problems in neural networks. The downsample module consists of two convolutional modules, and the upsample module consists of one convolutional layer and one deconvolutional layer. In the down_block and up_block, feature maps with the same width and height are concatenated and fused, achieving multi-scale feature fusion.

[0104] Step 2.4: The sketch vectorization module includes a differentiable rasterizer. The input of the sketch vectorization module is a pixel-level 3×256×256 single object sketch. For the input sketch, n Bézier curves are initialized. The position information of the control points of these curves and the color information of the lines are used as trainable parameters. The module continuously optimizes and iterates so that the visual effect of the final vectorized sketch composed of Bézier curves perfectly approximates the given pixel-level object sketch, thus achieving the purpose of vectorization. The output of the sketch vectorization module is a vectorized sketch of a single object.

[0105] Step 2.5: The scene sketch combination module, based on the multiple scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module, places the vectorized sketches of individual objects output by the sketch vectorization module into their corresponding positions, and finally outputs multiple scene sketches.

[0106] Step 3: Train the models for the scene layout modeling module and the single object sketch generation module constructed in Step 2;

[0107] Step 3.1: In the network structure corresponding to this invention, only the scene layout modeling module and the single object sketch generation module need to be trained.

[0108] For training the scene layout modeling module, AdamW was selected as the parameter optimizer, with an initial learning rate of 0.000288, a maximum number of iterations of 25, a batch size of 64 per iteration, and betas parameters of (0.9, 0.95).

[0109] The training of the single object sketch generation module is divided into two stages. The first stage trains the autoencoder model, using AdamW as the parameter optimizer with an initial learning rate of 0.0001, Betas parameters of (0.5, 0.9), a maximum number of iterations of 200, and a batch size of 4 per iteration. The second stage trains the conditional diffusion generation model, also using AdamW as the parameter optimizer with an initial learning rate of 0.0001 and Betas parameters of (0.5, 0.999).

[0110] Step 3.2: The loss function is used to evaluate the performance of the network model. The loss functions defined in this invention for different training stages are as follows:

[0111] For training the scene modeling module, the following cross-entropy loss function is used:

[0112]

[0113] Among them, o i This represents the value of the i-th layout element, o jThis represents the values ​​of all elements preceding the i-th element;

[0114] For the first stage of training in the single-object sketch generation module, there are three loss components:

[0115] L kl =D KL (z|N(z r ))

[0116] L l1 =||(inputs)-(recs)||

[0117] L perceptual =||Vgg(inputs)-Vgg(recs)||

[0118] Among them, D KL The KL divergence metric represents latent space coding and random Gaussian noise; L l1 It calculates the L1 distance between the input image and the reconstructed image, where inputs and recs represent the input image and the reconstructed image, respectively; L perceptual It calculates the L1 distance between the features of the input image and the reconstructed image after processing by the pre-trained VGG model, also known as perceptual loss; the final loss function of the first stage of training is a weighted sum of the parts:

[0119] L first =λ1·L l1 +λ2·L perceptual +λ3·L kl

[0120] Where λ1, λ2, and λ3 are taken as 1, 1, and 10, respectively. -6 ;

[0121] For the second stage of training of the single-object sketch generation module, the following loss function is used:

[0122] L second =E ε(x),C,f,r～N(0,1),t [||rr θ (z t ,t,C,f)|| 2 ]

[0123] Where C,f represents the category corresponding to a single object and the image features extracted by the feature extraction model, and r represents the real noise added in the current time step. θ Represents the Unet denoising model, r θ (z t ,t,C,f) represents the noise added at the current time step predicted by the denoising model, and the L2 distance between the two is used as the loss function for this stage;

[0124] Step 3.3: For the training of the two modules in this invention, during each iteration of model training, the scene image layout training set obtained in Step 1.1 is input into the scene layout modeling module, and the object sketch image training set obtained in Step 1.2 is input into the single object sketch generation module. The output of the scene layout modeling module and the true values ​​contained in the scene image layout training set obtained in Step 1.1 are input into the loss function to calculate the loss. The output of the single object sketch generation module and the true values ​​contained in the object sketch image training set obtained in Step 1.2 are input into the loss function to calculate the loss, and backpropagation is used to update the model parameters. The model file is saved every 20 iterations, and after training, the model files under different iterations are tested one by one. The program is run to sample and obtain the results. The results are compared to obtain the best performing model, and the trained scene layout modeling module and single object sketch generation module are obtained.

[0125] Step 4: Based on the scene image panoramic segmentation module, sketch vectorization module, and scene sketch combination module built in Step 2, as well as the scene layout modeling module and single object sketch generation module trained in Step 3, construct a complete scene sketch generation model.

[0126] like Figure 1 As shown, Figure 1 The training process for each module required for training in this invention and the complete model inference process are explained.

[0127] Step 4.1: Input the scene image into the panoramic scene image segmentation module. The panoramic segmentation network model in the panoramic scene image segmentation module loads the pre-trained model parameters. The input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0128] Step 4.2: Input the semantic layout of the original scene image output by the scene image panoramic segmentation module in Step 4.1 into the scene layout modeling module. Define the semantic layout of the original scene image in the panoramic segmentation result of the scene image as a vector as a representation of the layout information of the scene image, and input it into the scene layout modeling module. The L transformer layer networks in the scene layout modeling module predict the value of the layout information of the next position based on the value of the current position of the input vector and the value before that position. Finally, m new scene layouts are obtained from the original layout information, where m is a customizable input value.

[0129] Step 4.3: Connect the scene image panoramic segmentation module and the single object sketch generation module. Based on the semantic layout of the original scene image output by the panoramic segmentation network model in the scene image panoramic segmentation module, obtain the segmentation map of each foreground object. The feature extraction model in the single object sketch generation module extracts the features of the object image. The conditional fusion model fuses the category information in the semantic layout of the original scene image obtained in Step 2.1 and the features of the object image extracted by the feature extraction model into a single piece as the conditional input of the conditional diffusion generation model. The Unet denoising model combines random Gaussian noise and conditional... The output of the fusion module and the current time step t of the conditional diffusion generation model are used as input to predict the noise added between time step t and time step t-1. The predicted noise is subtracted from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1. The Unet denoising model is used again to predict the noise added between time step t-1 and time step t-2, using the predicted image, the output of the conditional fusion module, and time step t-1 as input, thus obtaining the predicted image of the previous time step again. This process is repeated until the initial time step is reached, resulting in a sampled sketch of a single object.

[0130] Step 4.4: Vectorize the single object sketch obtained in Step 4.3. Simultaneously, based on the multiple scene image layouts predicted from the semantic layout of the original scene image by the scene layout modeling module, organize the vectorized sketches of the single objects output by the sketch vectorization module onto the final canvas to form the final scene sketch output. For the placement of the foreground sketch, it can be placed according to the position information in the corresponding layout. For the placement of the background sketch, pixel clustering is performed on the background segmentation map in the input scene image output in Step 2.1, and the positions of K cluster centers are used as the final positions of the background sketch. Backgrounds of the same category can be placed at the K positions using the same generated sketch. Thus, multiple scene sketches with different scene layouts are obtained.

[0131] Model Inference

[0132] (1) Read the network model file that has been fully iterated and loaded the model parameters into the program model. Modify the model to inference mode and fix the parameters of each network layer.

[0133] (2) Import the input scene image into the program while keeping the original image size unchanged. The program first uses the scene image panoramic segmentation module to perform panoramic segmentation on the image to obtain the initial scene image semantic layout information. The scene layout modeling module takes the initial semantic layout as input and obtains multiple new scene layouts for subsequent use. The single object sketch generation module uses the object category and object image features as conditions to sample and obtain a single object sketch with a shape of 3×256×256. After passing through the sketch vectorization module, a single sketch in vector format is obtained. Finally, combined with the multi-scene layout information given by the scene layout modeling module, multiple scene sketches with the same size as the input scene image are given.

[0134] (3) The two methods compared with the results obtained in this invention are CLIPascene and Info-drawing, with the test set coming from the MS-COCO dataset. The performance of the model results is judged by the LPIPS, FID, and CLIP evaluation metrics, which can assess the similarity between images. As shown in Table 1, the arrows on the right side of each metric indicate that the larger the value, the higher the similarity between the two images, and vice versa. As can be seen from Table 1, the results obtained in this invention outperform the other two methods in all three metrics.

[0135] Table 1 Comparison of various indicators for different methods

[0136] method LPIPS (↓) FID(↓) CLIP(↑) CLIPascene 0.461 84.943 0.784 Info-drawing 0.512 73.059 0.755 Ours 0.340 24.354 0.863

[0137] The beneficial effects of this invention are as follows: This invention proposes a method for generating diverse hand-drawn sketches of complex scenes based on semantic layout. The design of the diffusion process in the latent space greatly shortens the model sampling time and improves the generation efficiency. This method not only realizes the transformation from complex scene images to sketches, but also generates scene sketches with more diverse layouts, richer object shapes, variable number of objects in the scene, and customizable sketch abstraction levels. Furthermore, the final generated sketches are closer to human hand-drawn sketches, making up for the shortcomings of existing scene sketch generation algorithms.

[0138] This invention also provides a scene hand-drawn sketch generation and editing system based on semantic layout, including:

[0139] The scene image preprocessing module is used to preprocess the scene image to obtain image sketch pairs of individual objects in the preprocessed scene image.

[0140] The scene image panoramic segmentation module is used to perform panoramic segmentation on the input scene image, convert the scene-level information of the scene image into object-level information, and output the semantic layout of the original scene image, the categories of all objects in the input scene image, and the corresponding object instance segmentation map and background segmentation map.

[0141] The scene layout modeling module is used to predict the semantic layout of the original scene image and generate multiple scene image layouts.

[0142] The single object sketch generation module is used to generate pixel-level single object sketches in the input scene image and realize the same object corresponding to various forms of object sketches.

[0143] The Sketch Vectorization module is used to convert pixel-level sketches of individual objects in a scene image into vectorized sketches of individual objects.

[0144] The Scene Sketch Combination Module is used to combine vectorized sketches of a single object into multiple scene sketches based on the layout of multiple generated scene images;

[0145] The model training module is used to train the models for the scene layout modeling module and the single object sketch generation module.

[0146] The scene sketch generation model building module is used to implement a scene image panoramic segmentation module, a sketch vectorization module, and a scene sketch combination module, as well as a trained scene layout modeling module and a single object sketch generation module, to build a complete scene sketch generation model.

[0147] The present invention also provides a scene hand-drawn sketch generation and editing device based on semantic layout, comprising:

[0148] Memory: A computer program that stores the above-mentioned method for generating and editing scene hand-drawn sketches based on semantic layout, and is a computer-readable device;

[0149] Processor: Used to implement the semantic layout-based scene hand-drawn sketch generation and editing method when executing the computer program.

[0150] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the aforementioned method for generating and editing scene hand-drawn sketches based on semantic layout.

Claims

1. A method for semantic layout based scene hand sketch generation and editing, characterized in that, The method comprises the following steps: Step 1, scene image preprocessing, obtaining the image sketch image pair of a single object in the preprocessed scene image; The specific method of step 1 is: Step 1.1, for a plurality of objects in the scene image and the sketch corresponding to the object, the object and the corresponding sketch are one-to-one corresponding, forming an image sketch pair of a single object, and the object sketch is taken as a real target image; Step 1.2, read the single object image sketch pair obtained in step 1.1 through the pytorch deep learning image processing library tool, adjust the shape of the single object image sketch pair to the size required by the single object sketch generation module before inputting the single object sketch generation module; at the same time, the single object image sketch pair is normalized, the input image data color value range is mapped from [0~255] integer value to [-1~1] floating point value, and the preprocessed single object image sketch image pair is obtained; Step 2, construct a scene image panorama segmentation module, a scene layout modeling module, a single object sketch generation module, a sketch vectorization module and a scene sketch combination module; wherein: The scene image panorama segmentation module is used for panorama segmentation of the input scene image, converting the scene level information of the scene image into object level information, outputting the semantic layout of the original scene image, the category of all objects in the input scene image and the corresponding object instance segmentation map and background segmentation map; The scene layout modeling module is used for predicting the semantic layout of the original scene image and generating a plurality of scene image layouts; The single object sketch generation module is used for generating pixel-level single object sketches in the input scene image and realizing a plurality of object sketches corresponding to the same object; The sketch vectorization module is used for converting the pixel-level single object sketch in the scene image into a single object vector sketch; The scene sketch combination module is used for combining the single object vector sketch into a plurality of scene sketches according to the generated plurality of scene image layouts; The specific method of step 2 is: Step 2.1, the scene image panorama segmentation module comprises a panorama segmentation network model, and the panorama segmentation network model is used for panorama segmentation of the input scene image, obtaining the semantic layout of the original scene image, the category of all objects in the input scene image and the corresponding object instance segmentation map and background segmentation map; Step 2.2, the scene layout modeling module includes a transformer architecture-based network model, the transformer architecture-based network model includes a plurality of transformer layers, each transformer layer is composed of two LayerNorm normalization layers, a causal self-attention layer and an mlp multi-layer perceptron; the mlp multi-layer perceptron is composed of a Linear layer, a GELU activation layer and a Linear layer; by inputting a starting layout information element in the semantic layout of the original scene image obtained in step 2.1, the transformer layer predicts the next possible scene layout information element, and finally outputs a plurality of scene image layouts predicted by the semantic layout of the original scene image; Step 2.3, the single object sketch generation module includes a self-encoding model and a conditional diffusion generation model, the conditional diffusion generation model inputs the class of all objects in the input scene image and the corresponding object instance segmentation map obtained in step 2.1, and the single object sketch generation module outputs a pixel-level single object sketch in the input scene image; Step 2.4, the sketch vectorization module includes a differentiable rasterizer, the input of the sketch vectorization module is the pixel-level single object sketch in the input scene image obtained in step 2.3, for the input sketch, the position information of the control points of the curves and the color information of the lines are initialized as trainable parameters, and the optimization iteration is continuously optimized so that the visual effect of the final vectorized sketch composed of the Bezier curves perfectly approximates the given pixel-level object sketch, achieving the purpose of vectorization, and the sketch vectorization module outputs a single object vectorized sketch; Step 2.5, the scene sketch combination module combines the plurality of scene image layouts predicted by the semantic layout of the original scene image output by the scene layout modeling module and the single object vectorized sketch output by the sketch vectorization module, and finally outputs a plurality of scene sketches; Step 3, model training is performed on the scene layout modeling module and the single object sketch generation module constructed in step 2; Step 4, based on the scene image panoramic segmentation module, the sketch vectorization module and the scene sketch combination module constructed in step 2, and the scene layout modeling module and the single object sketch generation module trained in step 3, a complete scene sketch generation model is constructed; The specific method of step 4 is: Step 4.1, input the scene image into the scene image panoramic segmentation module, the panoramic segmentation network model in the scene image panoramic segmentation module loads the pre-trained model parameters, the input size of the panoramic segmentation network model is the original size of the scene image, and the output is the panoramic segmentation result of the scene image, including the semantic layout of the original scene image, the class of all objects in the input scene image and the corresponding object instance segmentation map and background segmentation map; Step 4.2, input the semantic layout of the original scene image output by the scene image panorama segmentation module in step 4.1 into the scene layout modeling module, define the semantic layout of the original scene image in the panorama segmentation result of the scene image as a vector as the representation of the layout information of the scene image, and input it into the scene layout modeling module, the L transformer layer network in the scene layout modeling module predicts the value of the next bit of layout information according to the value of the current position and the value before the position, and finally obtains m new scene layouts from the original layout information, m is a customizable input value; Step 4.3, connect the scene image panorama segmentation module and the single object sketch generation module, obtain the segmentation map of each foreground object based on the semantic layout of the original scene image output by the panorama segmentation network model in the scene image panorama segmentation module; the feature extraction model in the single object sketch generation module extracts the features of the object image; the conditional fusion model fuses the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted using the feature extraction model into one as the conditional input of the conditional diffusion generation model; the Unet denoising model takes the random Gaussian noise, the output of the conditional fusion module and the current time step t of the conditional diffusion generation model as input, predicts the noise added between time step t and time step t-1, subtracts the predicted noise from the input random Gaussian noise to obtain the predicted image corresponding to time step t-1, uses the Unet denoising model again to take the predicted image, the output of the conditional fusion module and time step t-1 as input, predict the noise added between time step t-1 and time step t-2, and obtain the predicted image of the previous time step again. In this way, the single object sketch is obtained by sampling. Step 4.4, vectorize the single object sketch obtained in step 4.3, and at the same time, arrange the vectorized sketch of the single object output by the sketch vectorization module on the final canvas to form the final scene sketch output according to the multiple scene image layouts predicted from the semantic layout of the original scene image output by the scene layout modeling module. For the placement of the foreground sketch, place it according to the position information in the corresponding layout; for the placement of the background sketch, perform pixel point clustering on the background segmentation map in the input scene image output in step 2.1, and use the positions of the K clustering centers as the final positions of the background sketch. The same class of background uses the same generated sketch placed at the K positions; thereby obtaining multiple scene sketches with different scene layouts.

2. The method according to claim 1, wherein, The specific method of step 3 is: Step 3.1, for the training of the scene layout modeling module, select AdamW as the parameter optimizer, set the initial learning rate, the maximum number of iterations, the batch size of single iteration and the betas parameter; For the training of the single object sketch generation module, two stages of training are performed, the first stage trains the auto-encoding model, selects AdamW as the parameter optimizer, sets the initial learning rate, the maximum number of iterations, the batch size of a single iteration, and the betas parameters; the second stage trains the conditional diffusion generation model, sets the initial learning rate, the maximum number of iterations, the batch size of a single iteration, and the betas parameters; Step 3.2, for the training of the scene layout modeling module, the following cross-entropy loss function is used: where o i represents the value of the i-th layout element, o j represents the value of all elements before the i-th element; For the first stage training of the single object sketch generation module, there are three parts of loss, respectively: L kl = D KL (z | N(z r )) L l1 = ||(inputs) - (recs)|| L perceptual = ||Vgg(inputs)-Vgg(recs)|| where D KL represents the KL divergence measure of the latent space encoding and random Gaussian noise; L l1 is the L1 distance between the input image and the reconstructed image, inputs and recs represent the input image and the reconstructed image, respectively; L perceptual is the L1 distance between the features of the input image and the reconstructed image after passing through the pre-trained model VGG, also known as the perceptual loss; the final loss function of the first stage of training is the weighted sum of the parts: L first = λ1·L l1 + λ2·L perceptual + λ3·L kl wherein λ1, λ2, λ3 are taken as 1, 1, 10 respectively -6 ; For the second stage training of the single object sketch generation module, the following loss function is used: L second = E ε(x),C,f,r～N(0,1),t [||r-r θ (z t ,t,C,f)| 2 ] Wherein, C, f represents the category corresponding to a single object and the image feature extracted by the feature extraction model, r represents the real noise added in the current time step, r θ represents the Unet denoising model, r θ (z t , t, C, f) represents the noise added in the current time step predicted by the denoising model, and the L2 distance is taken as the loss function of this stage; Step 3.3, in each iteration process of model training, the semantic layout of the original scene image output by step 2.1 is input into the scene layout modeling module for calculation, the single object sketch image pair output by step 1.2 is input into the single object sketch generation module for calculation, and after each calculation, the model output and the true target image output by step 1.1 are input into the loss function to calculate the loss size, and the model parameters are updated by back propagation; save the model file, and after the training is completed, test the model files under different iteration numbers one by one, run the program to sample the results, and compare them to obtain the final optimal model, and obtain the trained scene layout modeling module and single object sketch generation module. 3.The method of claim 1, wherein, The auto-encoding model and the conditional diffusion generation model in step 2.3 specifically include: Step 2.3.1, auto-encoding model The auto-encoding model is an encoder-decoder structure network, the encoder part adopts Conv convolution layer, Downsample downsampling layer and GN normalization layer, which is used to encode the preprocessed single object sketch image pair obtained in step 1.2 to the latent vector in the latent space; the decoder part also adopts Conv convolution layer, Upsample upsampling layer and GN normalization layer, which is used to decode the latent vector in the latent space to the image in the pixel space; Step 2.3.2, conditional diffusion generation model The conditional diffusion generation model is composed of a feature extraction model, a condition fusion model and a Unet denoising model; The feature extraction model adopts the FPN feature pyramid network architecture, the bottom-up module in the feature extraction model adopts the Resnet50 structure, and the feature extraction model is used to extract the features of the object image; The condition fusion model is composed of a Linear layer and a Swish activation layer, which is used to fuse the class information in the semantic layout of the original scene image obtained in step 2.1 and the features of the object image extracted by the feature extraction model into a block as the condition input of the conditional diffusion generation model; The Unet denoising model is composed of a down_block, a mid_block and an up_block, and the three structures all include a basic Block module, wherein the down_block network structure includes a Downsample downsampling module, and the up_block network structure includes an Upsample upsampling module; the basic Block module is composed of two Linear linear layers and two Conv convolutional layers, a Swish activation function is used for the activation layer, the Downsample downsampling module is composed of two convolutional modules, and the Upsample upsampling module is composed of a convolutional layer and a deconvolutional layer; in the part where the width and height of the feature maps in the down_block and the up_block are the same, the feature maps are cascaded and fused, and multi-scale feature fusion is realized.

4. A semantic layout based scene hand sketch generation and editing system based on the method of any one of claims 1 to 3, characterized in that, Comprise: A scene image preprocessing module for preprocessing a scene image to obtain an image sketch image pair of a single object in the preprocessed scene image; A scene image panorama segmentation module for performing panorama segmentation on an input scene image, converting scene-level information of the scene image into object-level information, and outputting a semantic layout of the original scene image, a category of all objects in the input scene image, and a corresponding object instance segmentation map and background segmentation map; A scene layout modeling module for predicting the semantic layout of the original scene image and generating a plurality of scene image layouts; A single object sketch generation module for generating a pixel-level single object sketch in the input scene image and realizing a plurality of diverse object sketches corresponding to the same object; A sketch vectorization module for converting the pixel-level single object sketch in the scene image into a vectorized single object sketch; A scene sketch combination module for combining the vectorized single object sketch into a plurality of scene sketches according to the generated plurality of scene image layouts; A model training module for training the scene layout modeling module and the single object sketch generation module; A scene sketch generation model construction module for constructing a complete scene sketch generation model based on the scene image panorama segmentation module, the sketch vectorization module and the scene sketch combination module, and the trained scene layout modeling module and the single object sketch generation module.

5. A semantic layout based scene hand sketch generation and editing apparatus, characterized by, Comprise: A memory for storing a computer program of a scene hand-drawn sketch generation and editing method based on a semantic layout according to any one of claims 1-3, which is a computer-readable device; A processor for implementing the scene hand-drawn sketch generation and editing method based on a semantic layout according to any one of claims 1-3 when the computer program is executed.

6. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the scene hand-drawn sketch generation and editing method based on a semantic layout according to any one of claims 1-3.

Images