A method and device for generating line drawings corresponding to murals based on deep learning

Abstract

The present invention provides a method and device for generating line drawings corresponding to murals based on deep learning, the method comprising: collecting mural and line drawing image data, and performing preprocessing to form a real mural image I RB 、Real Line Drawing Image I RX ; The real mural image I RB 、Real Line Drawing Image I RX The RGB channels are used as inputs of the mural generator and the line drawing generator respectively; the mural generator and the line drawing generator generate murals and line drawings corresponding to the pixel level respectively to complete the style conversion. This scheme adopts the method of cyclic generative adversarial network to realize the style conversion between unpaired mural images and line drawing images, and introduces residual scale to prevent the instability of deep network training, which can minimize the loss of information and maintain the original texture structure information while converting more realistic style images.

Description

Technical Field

[0001] The present invention relates to the field of image processing, in particular to technologies related to image style conversion, and more particularly to a method and device for generating line drawings corresponding to murals based on deep learning. Background Art

[0002] As a non-renewable resource, ancient tomb murals will inevitably suffer from diseases due to many reasons such as the stability of their own materials, changes in the natural environment, and later human destruction. There are great difficulties in studying and protecting murals. The traditional method of manually repairing tomb murals relies on the rich experience and painting skills of scientific researchers, but it takes a long time and has extremely high requirements for the restorer. When digitally repairing and recording murals, the corresponding line drawings of the murals are often copied first. The restoration of digital murals can be assisted faster and better by completing and coloring the line drawings. The visual perception of the line drawings is simpler than the original images of the tomb murals, without redundant information, and is more conducive to showing the texture information of the murals and the actual degree of damage. However, there is a subjective problem in the artificially drawn line drawings. There may be deviations between the drawn line drawings and the tomb murals, and it is difficult to obtain the corresponding line drawings at the pixel level. The style conversion of the mural images by deep learning methods can obtain nearly corresponding line drawing images.

[0003] In many previous studies, paired images are needed for style transfer. In the task of mural style transfer, it is difficult to find highly matched line drawings, but there are many partially matched or unmatched line drawings. Therefore, in mural restoration, in order to facilitate the research work of mural restoration staff, how to effectively match and generate line drawings based on the style characteristics of murals needs further improvement and optimization. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating line drawings corresponding to murals based on deep learning. It can be trained under unpaired data to generate line drawings and murals corresponding to pixel levels.

[0005] Specifically, the present invention provides the following technical solutions:

[0006] On the one hand, the present invention provides a method for generating a line draft corresponding to a mural based on deep learning, the method comprising:

[0007] Step 1: Collect mural and line drawing image data and pre-process them to form a real mural image I RB 、Real Line Drawing Image I RX ;

[0008] Step 2: Transform the real mural image I RB 、Real Line Drawing Image I RXThe RGB channels are used as inputs of the mural generator and line drawing generator respectively;

[0009] Step 3: The mural generator and line draft generator generate murals and line drafts corresponding to the pixel level respectively to complete the style conversion.

[0010] Preferably, the structure of the mural generator and / or the line draft generator is:

[0011] It is composed of one or more densely connected residual blocks connected in series; the densely connected residual block structure is: a 3×3 convolution block is connected to a normalization layer, and the normalization layer is connected to a RELU activation function;

[0012] A residual scale is introduced into the densely linked residual block, and the residual scale is multiplied by the residual and then added to the main path. The residual scale has a value range of (0, 1).

[0013] Preferably, during the training, the mural generator and the line draft generator simultaneously construct a mural discriminator and a line draft discriminator;

[0014] Real mural image I RB As the input of the mural generator, the output generates the mural image I FB ; Real line drawing image I RX As the input of the line draft generator, the output generates the line draft image I FX ;

[0015] I RB And generate mural image I FB As the input of the mural discriminator, the output I RB and I FB The probability of being a real image; I RX And generate line drawing image I FX As the input of the line draft discriminator, the output I RX and I FX is the probability of being a real image.

[0016] Preferably, the discriminator is composed of one or more discriminant link groups connected in series, and the discriminant link group is composed of a 3×3 convolution block, a batch normalization layer and a leaky RELU activation function in series.

[0017] Preferably, the mural generator, line draft generator, mural discriminator, and line draft discriminator are trained to counter the loss function l GAN , content loss function as the objective function;

[0018] When training the mural discriminator and line draft discriminator, the corresponding mural generator and line draft generator are fixed, and the gradient ascent method is used to find l GANThe maximum value of the gradient is returned to update the parameters of the mural discriminator and the line draft discriminator until the training is completed after convergence. When training the mural generator and the line draft generator, the corresponding mural discriminator and the line draft discriminator are fixed, and the gradient descent method is used to find l GAN The minimum value of , the gradient is passed back to update the generator parameters until convergence and training is completed.

[0019] Preferably, the adversarial loss function is:

[0020]

[0021] Among them, D represents the processing of the discriminator, G represents the processing of the generator, and I F and I R represent the generated image and the real image respectively; or

[0022] The adversarial loss function is a least squares cross entropy function or a Wasserstein distance function.

[0023] Preferably, the content loss function is:

[0024]

[0025] Where W and H are the width and height of the image, respectively. x,y is the pixel value of the image at x, y, I R represents the real image, G A is the processing process of the line draft generator, G B represents the processing of the mural generator; or

[0026] The content loss function is the mean square error of the feature map in the mural discriminator / line draft discriminator.

[0027] Preferably, the mural discriminator and / or the line drawing discriminator is composed of one or more discriminant link groups connected in series; the discriminant link group is composed of a 3×3 convolution block, a batch normalization layer and a leaky RELU activation function connected in series.

[0028] Preferably, the mural discriminator and / or line drawing discriminator is a VGG16 network or a RESNET18 network.

[0029] On the other hand, the present invention also provides a device for generating line drafts corresponding to murals based on deep learning, which device includes at least: a processor and a memory, and the processor can call instructions stored in the memory to execute the method for generating line drafts corresponding to murals based on deep learning as described above.

[0030] In another aspect, the present invention further provides a system for generating line drawings corresponding to murals based on deep learning, the system comprising:

[0031] The image preprocessing module is used to collect mural and line drawing image data and perform preprocessing to form a real mural image I RB 、Real Line Drawing Image I RX ;

[0032] Mural generator, used to generate pixel-level murals and complete style conversion; real mural image I RB Taking RGB channels as input of the mural generator;

[0033] Line drawing generator, used to generate pixel-level line drawings and complete style conversion; real line drawing image I RX The RGB channels are used as input for the line drawing generator.

[0034] Preferably, when the mural generator and the line draft generator are being trained, the system further includes a mural discriminator and a line draft discriminator.

[0035] Preferably, the real mural image I RB As the input of the mural generator, the output generates the mural image I FB ; Real line drawing image I RX As the input of the line draft generator, the output generates the line draft image I FX ;

[0036] I RB And generate mural image I FB As the input of the mural discriminator, the output I RB and I FB The probability of being a real image; I RX And generate line drawing image I FX As the input of the line draft discriminator, the output I RX and I FX is the probability of being a real image.

[0037] Compared with the prior art, the technical solution of the present invention has at least the following advantages: This solution adopts a cyclic generative adversarial network to realize the style conversion between unpaired mural images and line draft images. In order to prevent the instability of training deep networks, a residual scale is introduced, and the residual is reduced by multiplying it by a constant from 0 to 1 before adding the residual to the main path. On this basis, dense links are used to ensure the detail processing effect of texture and style, and to avoid information loss as much as possible. At the same time, the adversarial loss function and the content loss function are used to control the network to maintain the original texture structure information while converting a more realistic style image. We use PSNR / SSIM to measure the similarity between the image after two style conversions and the original image, and NIQE to measure the look and feel of the image after one style conversion. Experiments have proved that the method proposed in the present invention has obvious advantages over other existing technologies (PIXTOPIX, MUNIT, cyclegan) in terms of the accuracy of reconstructed images (PSNR / SSIM) and the look and feel after style conversion (NIQE). BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a system structure diagram of an embodiment of the present invention;

[0040] Figure 2 A detailed structural diagram of a generator network according to an embodiment of the present invention;

[0041] Figure 3 is a detailed structural diagram of a discriminator network according to an embodiment of the present invention;

[0042] Figure 4 It is a comparative diagram of the test results of the embodiment of the present invention and the prior art;

[0043] Figure 5 4 is a flow chart of style conversion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0046] The present invention provides a method and system for generating corresponding line drawings of murals based on a generative adversarial network. Figure 1 , 2 As shown in Figure 3, the present invention adopts a densely linked residual block as the structure of the generator to extract deep features to prevent the instability of training deep networks and avoid the loss of texture and detail information. At the same time, the network is controlled by the adversarial loss function and the content loss function to maintain the original structural information while converting a more realistic style image. Compared with other existing technologies, the proposed method has obvious advantages in the accuracy of reconstructed images and the look and feel after style conversion. The following is a detailed description of the implementation steps of the technical solution of the present invention.

[0047] Step 1. Collect a certain number of murals and line drawing materials (paired or unpaired), and perform cropping, scaling, and flipping of the materials to the same specifications to enrich the sample data. In this embodiment, the data set uses digital images of the murals of the tomb of Prince Jiemin, the murals of the tomb of Princess Xincheng of Tang, and the murals of the tomb of Prince Yide. In addition, the image is segmented and cropped to 512×512 as the Ground truth, and slices of 48×48 area size are randomly selected each time to perform random flipping, translation, and rotation to enhance the data as the input of the network. The following continues to combine the above-mentioned embodiment data to elaborate on the technical solution of the present invention in detail.

[0048] Step 2: construct a mural discriminator and a line draft discriminator respectively, and construct two mural generators and a line draft generator. The mural generator and mural discriminator are used for generating and discriminating mural images; the line draft generator and line draft discriminator are used for generating and discriminating line draft images.

[0049] The real mural image I RB As the input of the mural generator, the real line drawing image I RX As input to the line drawing generator; RB And the generated mural image I FB As the input of the mural discriminator, I RX And the generated line drawing image I FX As the input of the line drawing discriminator.

[0050] The generator prevents the instability of training deep networks by constructing densely connected residual blocks and introduces residual scales, such as Figure 2 The constant α shown in , by adding the residual to the main path (adding the features extracted by the dense links to the features of the main path), multiplying it by a parameter α between 0 and 1 to reduce the residual. This process is to multiply the features extracted by the dense links by the constant α, and then return to the main path to form a residual. The parameter α here is set as a parameter during the network training process, and will be fixed to a specific constant during the use process after the training is completed. On this basis, dense links are used to ensure the detailed processing effect of texture and style, and to minimize the loss of information. The line drawing I generated by extracting and integrating the deep feature output through multiple densely linked residual blocks FX / Mural I FB image.

[0051] In the discriminator, we can use a cascade structure or a residual structure including an attention mechanism as a solution for deep feature extraction. In a more preferred embodiment, we use I RB and I FB As a mural discriminator D B The deep features are extracted through one or more discriminant link groups (a structure consisting of a 3×3 convolution block, a batch normalization layer, and a leaky RELU activation function), and the deep features are flattened and output through a sigmoid function. RB and I FB is the probability of a real image. RX and I FX As a line draft discriminator D A Input, output I RX and I FX is the probability of a real image. Of course, in addition to the preferred discriminator structure given above, we can also use classification networks such as VGG16 and RESNET18 as discriminators.

[0052] The specific operation process, detailed steps and operation mechanism of the generator and discriminator of the method for generating corresponding line drawings from murals based on the generative adversarial network in this embodiment are as follows:

[0053] The real mural I RB / Line Draft I RX The image is used as the line drawing G according to the RGB channel A / Mural G B Generator input. In a preferred embodiment, the structure of the mural / line drawing generator can adopt the same structure. Of course, it can also be implemented by using networks with different network structures, which is limited only by being able to realize its corresponding basic functions.

[0054] In a preferred embodiment, the generator can be specifically composed of a 3×3 convolution block (i.e. Figure 2 The structure of the generator is composed of a densely connected residual block composed of a Conv layer and a RELU activation function, with a batch normalization layer in the middle to stabilize network training, and no pooling layer to prevent pixel-level information from being removed. Figure 2 , consists of one or more densely connected residual blocks, each of which has a structure of a 3×3 convolution block connected to a normalization layer, and a normalization layer connected to a RELU activation function. When there are multiple densely connected residual blocks, the residual blocks are connected in series.

[0055] The generator will I RB / I RX The image is read in as RGB three channels, first converted to 64 channels through the convolution block, and then the feature depth is extracted and integrated through one or more subsequent densely linked residual blocks. The densely linked residual blocks are composed of dense links. The features extracted by multiple densely linked residual blocks are added to the features on the backbone in a residual manner (for example Figure 2 , where there are 5 densely connected residual blocks), and multiple times (e.g. Figure 2 After this operation, the 3-channel (RGB) I FX / I FB image.

[0056] In a preferred embodiment, the discriminator is a binary classifier based on deep learning. In a preferred embodiment, the structure of the mural / line drawing discriminator can adopt the same structure. Of course, it can also be implemented by a network with a different network structure, which is only limited to being able to realize its corresponding basic functions.

[0057] In a more preferred embodiment, reference Figure 3 As shown in FIG. 1 , the structure of the discriminator can be a structure composed of one or more discriminative link groups connected in series. The discriminative link group is composed of a 3×3 convolution block, a batch normalization layer, and a leakyRELU activation function connected in series.

[0058] Will I RX / I RB and the generated I FX / I FBAccording to the RGB three-channel input, the deep feature extraction is carried out through a series structure composed of several 3×3 convolution blocks, batch normalization layers and leaky RELU activation functions. After the deep features are flattened, the probability of the line drawing / mural image being real is output through the sigmoid function. Both the line drawing discriminator and the mural discriminator are only used during training to assist the training of the generator. After the training is completed, the line drawing generator and the mural generator are used to perform image style conversion. The flattening process here is to convert the n feature maps into n 1×1 feature maps through pooling, and then pull them up to an n-dimensional vector representation.

[0059] Step 3: Train the constructed mural generator, line drawing generator, mural discriminator, and line drawing discriminator. In the process of training the generator, it is required that the generated mural image is judged as a real image by the mural discriminator as much as possible, and the generated line drawing image is judged as a real image by the line drawing discriminator as much as possible. At the same time, it is required that the generated mural / line drawing is used as input and the line drawing / mural obtained by the line drawing / mural generator is compared with the original input I RX / I RB As close as possible.

[0060] In a more preferred embodiment, during the training process, we use two loss functions as our training objectives, namely, the adversarial loss function and the content loss function.

[0061] During the training process, the discriminator needs to judge the authenticity of the input image to assist the training of the generator. In the authenticity judgment, in a preferred embodiment, the adversarial loss function is used. Measures the JS divergence between the real image distribution and the generated image distribution, where D represents the processing of the discriminator, G represents the processing of the generator, and I F and I R Represent the generated image and the real image respectively, and E represents the expectation. When training the discriminator, it is expected that the discriminator will I R If it is judged as true (the output is close to 1), I F It is judged to be false (the output is close to 0). Therefore, when training the discriminator, the generator is fixed and the gradient ascent method is used to find l GAN The maximum value of the gradient is returned to update the discriminator parameters; when training the generator, the discriminator is fixed and the gradient descent method is used to find l GAN The minimum value of , the gradient is passed back to update the generator parameters.

[0062] Here, it should be noted that, in addition to the preferred adversarial loss function given in this embodiment, the loss function here can also adopt, for example, least squares cross entropy (LSGAN), Wasserstein distance (WGAN), etc. These conventional replacements should be considered to fall within the scope of protection of the present invention.

[0063] In the training of the generator, the real mural / line drawing image is used as the input of the line drawing / mural generator, and the generated line drawing / mural image is used as the input of the mural / line drawing generator. The generated mural / line drawing image should be as close as possible to the original input real mural / line drawing image to ensure that the information loss is reduced in the process of style transfer. In a more preferred embodiment, we use the reconstruction loss at the image pixel level to measure this process. The content loss function Control the style to reduce information loss during the cyclic conversion process. W and H are the width and height of the image, respectively, and I x,y is the pixel value of the image at x, y, I R represents the real image, G A is the processing process of the line draft generator, G B Represents the processing of the mural generator. MSE At the same time, the reconstructed image is closer to the original image, indicating that less information is lost during the style transfer process.

[0064] Here, it should be noted that in addition to the preferred content loss function given in this embodiment, the mean square error of the feature map in the discriminator can also be used as the content loss function. These conventional replacements should be regarded as falling within the scope of protection of the present invention.

[0065] In a more preferred embodiment, the ratio of the calculated values ​​of the adversarial loss function and the content loss function is 1:400.

[0066] Step 4: After the training of the discriminator and generator models converge, the mural and line drawing generator can be used for style conversion, taking the real mural / line drawing as input to generate the corresponding line drawing / mural at the pixel level. After the network training is completed, the two discriminators do not participate in the generation of images and the use of subsequent models. The images to be converted are manually selected as the input of the line drawing or mural generator, and the corresponding style images are output.

[0067] Combination Figure 5 As shown in the figure, after the training is completed, when the generator is used to perform image style conversion, the discriminator is no longer used, and the steps are simplified as follows:

[0068] Step 1: Collect mural and line drawing image data and pre-process them to form a real mural image I RB 、Real Line Drawing Image I RX ;

[0069] Step 2: Transform the real mural image I RB 、Real Line Drawing Image I RX The RGB channels are used as inputs of the mural generator and line drawing generator respectively;

[0070] Step 3: The mural generator and line draft generator generate murals and line drafts corresponding to the pixel level respectively to complete the style conversion.

[0071] In another specific embodiment, the present invention further elaborates on the content of the scheme of the present invention by comparing with the experimental results of the prior art. In this embodiment, the hardware and software environment of the experiment can be set as follows: the software environment of this embodiment is windos, implemented in pytorch1.7, the CPU is 10700k, the frequency is 3.8MHZ, the running memory size is 64GB, the frequency is 3200MHZ, the GPU is NVIDIA GeForce GTX3080 (10GB), and the hard disk is Samsung 970EVOPlus NVMe M.2 (1TB). In the experiment, the Adam optimizer with a momentum parameter of 0.9 is used to optimize the model parameters, the initial value of the learning rate is set to 0.01, and the learning rate is adjusted every 50 rounds. The specific experimental content is:

[0072] 1. Explore the impact of different generator structures on network performance. In the training of unpaired image style transfer, maintaining consistent structural information is a challenging problem. We use densely linked residual blocks to maintain consistent structural information as the network depth continues to deepen. We increase the number of densely linked residual blocks from 2 to 8, and the image quality (PSNR) reconstructed after cyclic style transfer also increases almost linearly from 15.2db to 25.7. After that, the image reconstruction quality does not change much as the number gradually increases to 16, while the parameter memory increases by 83%. Therefore, in subsequent comparative experiments, we lock the number of densely linked residual blocks at 8.

[0073] 2. Comparison with existing advanced methods (cyclegan, MUNIT, PIXTO PIX) is shown in Table 1:

[0074] Table 1

[0075]

[0076] The method proposed in this paper has achieved obvious advantages in image reconstruction quality (PSNR / SSIM) and perception (NIQE). Figure 4 The performance of different methods in the task of converting murals into line drawing styles is demonstrated.

[0077] In another aspect, the technical solution of the present invention can also be implemented in a system manner, the system comprising:

[0078] The image preprocessing module is used to collect mural and line drawing image data and perform preprocessing to form a real mural image I RB、Real Line Drawing Image I RX ;

[0079] Mural generator, used to generate pixel-level murals and complete style conversion; real mural image I RB Taking RGB channels as input of the mural generator;

[0080] Line drawing generator, used to generate pixel-level line drawings and complete style conversion; real line drawing image I RX The RGB channels are used as input for the line drawing generator.

[0081] Preferably, when the mural generator and the line draft generator are being trained, the system further includes a mural discriminator and a line draft discriminator.

[0082] Preferably, the real mural image I RB As the input of the mural generator, the output generates the mural image I FB ; Real line drawing image I RX As the input of the line draft generator, the output generates the line draft image I FX ;

[0083] I RB And generate mural image I FB As the input of the mural discriminator, the output I RB and I FB The probability of being a real image; I RX And generate line drawing image I FX As the input of the line draft discriminator, the output I RX and I FX is the probability of being a real image.

[0084] In another embodiment, the present solution can be implemented by means of a device, which may include a corresponding module for performing each or several steps in each of the above-mentioned embodiments. Therefore, each step or several steps of each of the above-mentioned embodiments may be performed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specifically configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0085] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiment of the present solution includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present solution. The processor performs the various methods and processes described above. For example, the method implementation in the present solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0086] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0087] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for generating line drawings corresponding to murals based on deep learning, characterized in that: The method comprises: Step 1: Collect mural and line drawing image data and pre-process them to form a real mural image , Real line drawing images ; Step 2: Convert the real mural image , Real line drawing images The RGB channels are used as inputs of the mural generator and line drawing generator respectively; Step 3: The mural generator and line draft generator generate murals and line drafts corresponding to the pixel level respectively to complete the style conversion; The structure of the mural generator and / or the line draft generator is: It is composed of one or more densely connected residual blocks connected in series; the densely connected residual block structure is: a 3×3 convolution block is connected to a normalization layer, and the normalization layer is connected to a RELU activation function; A residual scale is introduced into the densely linked residual block, and the residual scale is multiplied by the residual and then added to the main path. The residual scale has a value range of (0, 1); During the training of the mural generator and the line draft generator, a mural discriminator and a line draft discriminator are simultaneously constructed; Real mural image As the input of the mural generator, the output generates a mural image ; Real line drawing image As the input of the line drawing generator, the output generates a line drawing image ; and generate mural images As the input of the mural discriminator, the output and The probability of being a real image; and generate line art images As the input of the line draft discriminator, the output and The probability of being a real image; The mural generator, line draft generator, mural discriminator, and line draft discriminator are trained to counter the loss function , content loss function as the objective function; When training the mural discriminator and line draft discriminator, the corresponding mural generator and line draft generator are fixed, and the gradient ascent method is used to find The maximum value of the gradient is returned to update the parameters of the mural discriminator and the line draft discriminator until the training is completed after convergence. When training the mural generator and the line draft generator, the corresponding mural discriminator and the line draft discriminator are fixed, and the gradient descent method is used to find The minimum value of , the gradient is passed back to update the generator parameters until convergence and training is completed.

2. The method according to claim 1, characterized in that The discriminator is composed of one or more discriminant link groups connected in series, and the discriminant link group is composed of a 3×3 convolution block, a batch normalization layer and a leaky RELU activation function connected in series.

3. The method according to claim 1, characterized in that The adversarial loss function is: in, represents the processing of the discriminator, Represents the processing of the generator, and represent the generated image and the real image respectively.

4. The method according to claim 1, characterized in that: The content loss function is: Where W and H are the width and height of the image respectively. is the pixel value of the image at x,y, represents the real image, This is the processing of the line draft generator. Represents the processing of the mural generator.

5. The method according to claim 1, characterized in that The mural discriminator and / or line draft discriminator is a VGG16 network or a RESNET18 network.

6. A device for generating line drawings corresponding to murals based on deep learning, characterized in that: The device at least includes: a processor and a memory, and the processor can call instructions stored in the memory to execute the method for generating a line draft corresponding to a mural based on deep learning as described in any one of claims 1-5.

Images