Image processing model training, image processing method, device, equipment and medium

Abstract

The embodiment of the present invention discloses an image processing model training, image processing method, device, equipment and medium. Specifically, it includes: obtaining a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed; inputting the line draft image to be processed and the pixel point image to be processed into an image generation module, and generating a line draft coloring image to be verified through the image generation module; obtaining the original sample image of the line draft image to be processed, inputting the original sample image and the line draft coloring image to be verified into an image discrimination module, and generating image authenticity data through the image discrimination module; determining the loss value of the image processing model according to the image authenticity data and the labeled data that matches the image authenticity data, and training the image processing model according to the loss value. The technical solution of the embodiment of the present invention can eliminate the checkerboard effect of the automatically generated line draft coloring image, thereby improving the image quality of the automatically generated line draft coloring image.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to image processing model training, image processing methods, devices, equipment and media. Background Art

[0002] With the development of science and technology, image processing has become a widespread and fundamental problem. Existing image processing mainly uses image processing methods based on deep learning. The image processing method based on deep learning is to train the image processing model through a large number of image sets, and adjust the image processing model and model parameters to achieve the purpose of generating high-quality images.

[0003] However, existing deep learning-based image processing methods easily cause the color of a certain part of the generated image to be darker than other parts, which means that it is easy to cause a checkerboard effect in the generated image. Summary of the invention

[0004] Embodiments of the present invention provide an image processing model training, an image processing method, an apparatus, a device and a medium, which can eliminate the checkerboard effect of an automatically generated line drawing colored image, thereby improving the image quality of the automatically generated line drawing colored image.

[0005] According to one aspect of the present invention, there is provided an image processing model training method, comprising:

[0006] Acquire a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed;

[0007] Inputting the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model, so as to generate a line draft colored image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer;

[0008] Acquire an original sample image of the line draft image to be processed, and input the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module;

[0009] A loss value of the image processing model is determined according to the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value.

[0010] According to another aspect of the present invention, there is provided an image processing method, comprising:

[0011] Get the line drawing image to be processed;

[0012] Performing brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed; the brush stroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed;

[0013] Inputting the brushstroke image to be processed into an image generation module of a trained image processing model, so as to perform image processing on the brushstroke image to be processed through the image generation module to obtain a target line drawing colored image;

[0014] The target line drawing colored image is subjected to noise filtering processing by using a joint bilateral filter to obtain a target image.

[0015] According to another aspect of the present invention, there is provided an image processing model training device, comprising:

[0016] An image acquisition module, used to acquire a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed;

[0017] A line draft coloring image generation module to be verified, used for inputting the line draft image to be processed and the pixel point image to be processed into the image generation module of the image processing model, so as to generate the line draft coloring image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer;

[0018] An image authenticity data generation module, used for obtaining an original sample image of the line draft image to be processed, and inputting the original sample image and the line draft colored image to be verified into the image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module;

[0019] An image processing model training module is used to determine the loss value of the image processing model based on the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value.

[0020] According to another aspect of the present invention, there is provided an image processing apparatus, comprising:

[0021] A module for acquiring line draft images to be processed, used for acquiring line draft images to be processed;

[0022] A brushstroke image acquisition module to be processed is used to perform brushstroke processing on the line draft image to be processed to obtain a brushstroke image to be processed; the brushstroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed;

[0023] A target line draft colored image generation module is used to input the brush stroke image to be processed into the image generation module of the trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generation module to obtain a target line draft colored image;

[0024] The target image generation module is used to perform noise filtering processing on the target line draft colored image through a joint bilateral filter to obtain a target image.

[0025] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0026] one or more processors;

[0027] A storage device for storing one or more programs;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing model training method provided on the one hand, or the image processing method provided on the other hand.

[0029] According to another aspect of the present invention, a computer storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the image processing model training method provided on the one hand, or the image processing method provided on the other hand is implemented.

[0030] The embodiment of the present invention obtains a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed, inputs the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model, so as to generate a line draft colored image to be verified through the image generation module, and obtains an original sample image of the line draft image to be processed, inputs the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module, thereby determining a loss value of the image processing model according to the image authenticity data and labeled data that matches the image authenticity data, and then training the image processing model according to the loss value, so as to perform image processing on the line draft image to be processed using the image generation module of the successfully trained image processing model, thereby solving the problem of checkerboard effect in the generated image caused by the existing image processing method, and being able to eliminate the checkerboard effect of the automatically generated line draft colored image, thereby improving the image quality of the automatically generated line draft colored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flowchart of an image processing model training method provided in Embodiment 1 of the present invention;

[0033] Figure 2 is an example flow chart of an image processing model training method provided in Embodiment 1 of the present invention;

[0034] Figure 3 is a flow chart of an image processing method provided by Embodiment 2 of the present invention;

[0035] Figure 4 is an example flow chart of an image processing method provided by Embodiment 2 of the present invention;

[0036] Figure 5 is a schematic diagram of an image processing model training device provided in Embodiment 3 of the present invention;

[0037] Figure 6 is a schematic diagram of an image processing device provided by a fourth embodiment of the present invention;

[0038] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0040] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0041] Embodiment 1

[0042] Figure 1This is a flowchart of an image processing model training method provided by the first embodiment of the present invention. This embodiment is applicable to the case of eliminating the checkerboard effect of automatically generated line drawing coloring images. The method can be executed by an image processing model training device, which can be implemented by software and / or hardware, and can generally be directly integrated into an electronic device that executes this method. The electronic device can be a terminal device or a server device. The embodiment of the present invention does not limit the type of electronic device that executes the image processing model training method. Specifically, Figure 1 As shown, the image processing model training method may specifically include the following steps:

[0043] S110, obtaining a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed.

[0044] The line draft image to be processed may be any binary grayscale image outlined by lines and shadows that needs to be processed. The pixel point image to be processed may be an image composed of pixel points in a color image corresponding to the line draft image to be processed.

[0045] In an embodiment of the present invention, a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed are obtained, so as to input the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model. It should be noted that the embodiment of the present invention does not limit the specific implementation method of obtaining the line draft image to be processed and the pixel point image to be processed, as long as the acquisition of the line draft image to be processed and the pixel point image to be processed can be achieved. It is understandable that the line draft image to be processed can be colored according to the pixel point image to be processed to obtain a color image corresponding to the line draft image to be processed, thereby effectively improving the visual perception of the animation and enhancing the expressiveness of the picture.

[0046] Optionally, before obtaining the line draft image to be processed and the pixel point image to be processed that matches the line draft image to be processed, the process may also include: obtaining the original sample image; performing edge extraction on the original sample image through an edge extraction model to obtain the line draft image to be processed; performing pixel point extraction on the original sample image through a pixel point extraction model to obtain the pixel point image to be processed.

[0047] Among them, the original sample image can be a color image with various colors that can be used as a sample image. The edge extraction model can be a model for extracting image edges, such as a Canny (edge ​​detection algorithm) model, or an XDoG (eXtended difference-of-Gaussians, an algorithm for generating edge images) model, etc., and the embodiments of the present invention do not limit this. The pixel point extraction model can be a model for extracting image pixels, such as an ORB (Oriented Fast and Rotated Brief, feature point extraction algorithm) model, or a SIFT (Scale-invariant feature transform, scale-invariant feature transform algorithm) model, etc., and the embodiments of the present invention do not limit this.

[0048] Specifically, before obtaining the line draft image to be processed and the pixel point image to be processed that matches the line draft image to be processed, the original sample image can be obtained, and the edge extraction model can be used to perform edge extraction on the original sample image to obtain the line draft image to be processed, and the pixel point extraction model can be used to perform pixel point extraction on the original sample image to obtain the pixel point image to be processed.

[0049] S120. Input the line draft image to be processed and the pixel point image to be processed into the image generation module of the image processing model, so as to generate the line draft colored image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer.

[0050] Among them, the image processing model can be a model for processing images, for example, it can be a GAN (generative adversarial network) network model, etc., and the embodiment of the present invention is not limited to this. The image generation module can be a module that can generate images in a model for processing images, for example, it can be a generator in a GAN network model, etc., and the embodiment of the present invention is not limited to this. The line draft coloring image to be verified can be an image generated by the image generation module after the line draft image is colored and processed, which needs to be verified. The target deconvolution layer can be the target layer of the deconvolution layer in the full convolution structure, for example, it can be the first deconvolution layer and the second deconvolution layer of the deconvolution layer, etc., and the embodiment of the present invention is not limited to this.

[0051] In an embodiment of the present invention, after obtaining the line draft image to be processed and the pixel point image to be processed that matches the line draft image to be processed, the line draft image to be processed and the pixel point image to be processed can be further input into the image generation module of the image processing model to generate the line draft coloring image to be verified through the image generation module. Specifically, the image generation module of the image processing model may include a full convolution structure, and the target deconvolution layer of the full convolution structure may include an upsampling layer and a convolution layer. It is understandable that the full convolution structure may include a convolution layer and a deconvolution layer. Exemplarily, the full convolution structure may include 5 convolution layers and 4 deconvolution layers.

[0052] In the prior art, the convolution kernel size of the deconvolution of the image generation module of the image processing model cannot always be divided by the step size, resulting in uneven overlapping in the deconvolution process, also known as the checkerboard effect.

[0053] The above technical solution adopts an upsampling plus convolution structure instead of deconvolution through the target deconvolution layer, can separate the upsampling part from the convolution, use the difference method to change the image size, and then perform convolution processing, thereby suppressing the generation of high-frequency artifacts and eliminating the checkerboard effect of the image.

[0054] Optionally, before generating the line draft colored image to be verified through the image generation module, it can also include: extracting features of the line draft image to be processed through a Sketch-a-Net (multi-channel deep neural network) model to obtain the line draft image features to be processed of the line draft image to be processed; acquiring the generated image features generated by the image generation module through the image generation module, and inputting the line draft image features to be processed into the image generation module, and performing feature splicing of the line draft image features to be processed and the generated image features through the image generation module.

[0055] The Sketch-a-Net model may be a pre-trained feature extraction model that can be used to extract features from the line drawing image to be processed. The line drawing image features to be processed may be features from the line drawing image to be processed extracted by the Sketch-a-Net model. The generated image features may be features from the line drawing image to be processed obtained by the image generation module.

[0056] Specifically, before generating a line draft colored image to be verified through an image generation module, feature extraction can be performed on the line draft image to be processed through a Sketch-a-Net model to obtain line draft image features to be processed of the line draft image to be processed, and features of a generated image generated by the image generation module can be obtained through the image generation module, and the line draft image features to be processed can be input into the image generation module, so that the line draft image features to be processed and the features of the generated image can be feature spliced ​​through the image generation module.

[0057] The above technical solution, by using the Sketch-a-Net model to extract the features of the line draft image to be processed, and by using the image generation module to perform feature splicing of the features of the line draft image to be processed and the features of the generated image, can solve the problem that the number of network layers of the image processing model is too shallow and the amount of data is too small, resulting in insufficient training of the image processing model network, thereby improving color overflow and image blur.

[0058] Optionally, before generating the line drawing colored image to be verified through the image generation module, it may also include: connecting the preset convolution layer of the full convolution structure of the image generation module with the preset deconvolution layer through a long connection method.

[0059] Among them, the preset convolution layer can be a convolution layer in a preset full convolution structure, for example, it can be the first convolution layer, it can also be the second convolution layer, etc., and the embodiment of the present invention does not limit this. The preset deconvolution layer can be a deconvolution layer in a preset full convolution structure, for example, it can be the first deconvolution layer, it can also be the second deconvolution layer, etc., and the embodiment of the present invention does not limit this.

[0060] Specifically, before the image generation module generates the line drawing colored image to be verified, the preset convolution layer of the full convolution structure of the image generation module can be connected to the preset deconvolution layer through a long connection. Exemplarily, the symmetrical convolution layer and deconvolution layer in the full convolution structure of the image generation module can be connected through a long connection.

[0061] S130, obtaining an original sample image of the line draft image to be processed, and inputting the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module.

[0062] The image discrimination module may be a module in a model for processing an image that can discriminate an input image, for example, a discriminator in a GAN network model, etc., and the embodiment of the present invention does not limit this. The image authenticity data may be probability data that the input image is a real image. It is understandable that the more real the image input to the image discrimination module is, the closer the image authenticity data is to 1.

[0063] In an embodiment of the present invention, after the line draft image to be processed and the pixel point image to be processed are input into the image generation module of the image processing model, and the line draft colored image to be verified is generated by the image generation module, the original sample image of the line draft image to be processed can be further obtained, and the original sample image and the line draft colored image to be verified are input into the image discrimination module of the image processing model, so as to generate image authenticity data by the image discrimination module. Exemplarily, the image discrimination module of the image processing model can include a fully connected layer and 1 / 2 of the convolution layer in the fully convolution structure of the image generation module.

[0064] S140. Determine a loss value of the image processing model according to the image authenticity data, the labeled data matching the image authenticity data, the original sample image, and the line draft colored image to be verified, so as to train the image processing model according to the loss value.

[0065] The label data may be data for labeling the image before the image is input into the image discrimination module. Exemplarily, if the original sample image is a real image, the label data of the original sample image may be 1. If the line drawing coloring image to be verified is a generated image, the label data of the line drawing coloring image to be verified may be 0. The loss value may be the loss data in the process of processing the image by the image processing model.

[0066] In an embodiment of the present invention, after the original sample image and the line drawing colored image to be verified are input into the image discrimination module of the image processing model and the image authenticity data is generated by the image discrimination module, the loss value of the image processing model can be further determined according to the image authenticity data, the labeled data matching the image authenticity data, the original sample image and the line drawing colored image to be verified, so as to train the image processing model according to the loss value. It can be understood that the smaller the loss value of the image processing model, the closer the image generated by the image generation module of the image processing model is to the real image, that is, the better the coloring effect of the image generated by the image generation module of the image processing model is.

[0067] Optionally, the loss value of the image processing model is determined based on the image authenticity data, labeled data matching the image authenticity data, the original sample image and the line draft colored image to be verified, including: determining the adversarial loss value of the image processing model based on the image authenticity data and the labeled data matching the image authenticity data; determining the L1 loss value and perceptual loss value of the image processing model based on the original sample image and the line draft colored image to be verified; determining the total variation loss value of the image processing model based on the line draft colored image to be verified; and determining the loss value through the adversarial loss value, L1 loss value, perceptual loss value and total variation loss value.

[0068] Among them, the adversarial loss value can be the difference between the labeled data matched by the image authenticity data and the image authenticity data. The L1 loss value can be the difference between each pixel of the original sample image and the line draft coloring image to be verified. The total variation loss value can be the difference between the pixel value of a certain pixel of the line draft coloring image to be verified and the pixel values ​​of its surrounding pixels. The perceptual loss value can be the difference between the multi-dimensional feature values ​​of the original sample image and the line draft coloring image to be verified.

[0069] Specifically, after the original sample image and the line draft colored image to be verified are input into the image discrimination module of the image processing model and the image authenticity data is generated by the image discrimination module, the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value of the image processing model can be further determined according to the image authenticity data and the labeled data that matches the image authenticity data, so as to determine the loss value by the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value, thereby training the image processing model according to the loss value. Optionally, the loss value is determined by the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value, which can be determined by the weighted average of the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value, or by the arithmetic average of the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value, etc., and the embodiments of the present invention do not limit this.

[0070] In the prior art, there are two generally accepted basic assumptions in image colorization: first, if adjacent pixels have similar brightness, their colors will also be similar; second, if adjacent pixels have similar features (such as texture and shape), their colors will also be similar. Currently, almost all colorization algorithms are based on these two assumptions.

[0071] Existing image colorization methods are mainly divided into the following categories: color diffusion based on manual brushstrokes, color transfer based on reference images, and methods based on deep learning. The color diffusion algorithm based on manual brushstrokes adds color brushstrokes to various areas of the grayscale image, diffuses the brushstrokes, and stops at the boundaries, and finally obtains a completely colored image. This method has high requirements on the input form of brushstrokes, but does not require a reference image and can adjust the coloring. The color transfer method based on reference images first needs to find an image with content similar to the image to be colored. When coloring, find the pixel in the reference image that is most similar to the current pixel to be colored, and use its color as the color of the current pixel. This method has high time complexity, and because of the pixel-by-pixel color transfer, it is easy to cause the problem of inconsistent color in the neighborhood space. In addition, a reference image is required, which is suitable for video sequence coloring. The method based on deep learning is to train the model through a large number of image sets, and to achieve the purpose of generating high-quality colored images by adjusting the network model and parameters. This method requires a large number of data sets as training data, but there is currently no data set that contains both color images and their corresponding line draft images. At the same time, the network architecture and the design of the loss function are the key to the quality of coloring, and they need to be adjusted repeatedly. However, the trained network can achieve almost instant colorization when used, and many current works have shown that the use of deep learning methods can achieve better colorization effects than traditional methods.

[0072] At present, deep learning-based methods such as the deepColor algorithm are built based on the GAN network model. It divides coloring into two stages: one is the color prediction stage and the other is the color rendering stage. The color prediction stage receives the line drawing image as input and outputs the predicted color information; the color rendering stage receives the line drawing image and the color image output in stage 1 as input and outputs the final coloring image. However, the colorized image output by this method has poor quality and the color of the colorized image may be biased.

[0073] In a specific example, the image processing model in the deep learning-based method can be a GAN network model, wherein the GAN network model is trained jointly by a generator and a discriminator network in a way that the two networks compete with each other; the generator is responsible for generating fake samples, and the discriminator is responsible for distinguishing between real samples and fake samples; the fake samples generated by the trained generator can achieve the purpose of making the fake look real. Figure 2 is an example flow chart of an image processing model training method provided by the first embodiment of the present invention. Figure 2 As shown, the method may specifically include the following contents:

[0074] An edge extraction algorithm is used to perform edge detection on a color image (i.e., the original sample image) to generate an edge map, and the edge map is used as a line drawing image to be processed to train the GAN network model; wherein the edge extraction algorithm can be Canny, DoG (difference-of-Gaussians, an algorithm for generating edge images) or XDoG, etc. A pixel point extraction algorithm is used to extract pixels from a color image to obtain a pixel point image to be processed.

[0075] The extracted line draft image to be processed and the pixel point image to be processed are input into the generator G (i.e., the image generation module) of the GAN network model to generate the line draft coloring image to be verified through the generator G. The generator G of the GAN network model is a fully convolutional structure, with a total of nine layers except for the input and output, and is connected by long connections between symmetrical layers in imitation of the U-Net (image segmentation network model) network model; and the first two layers of the deconvolution stage adopt an upsampling plus convolution structure to eliminate the checkerboard effect in the colored image.

[0076] The features of the line draft image to be processed are extracted through the pre-trained model Sketch-a-Net feature extractor, and the features of the line draft image to be processed are input into the generator G of the GAN network model and spliced ​​with the third layer features of the generator G.

[0077] The color image and the line drawing coloring image to be verified generated by the generator G are input to the discriminator D (i.e., the image discrimination module) of the GAN network model to determine whether the input image is a real image through the discriminator D. The discriminator D uses half of the convolutional layer of the generator G and adds a fully connected layer at the end to output a score to indicate the probability that the input image is a real image.

[0078] The GAN network model is trained by guiding its adversarial loss, L1 loss, total variation loss and perceptual loss to improve the image coloring processing of the GAN network model generator G.

[0079] Optionally, the adversarial loss function can be:

[0080]

[0081] Among them, x represents the real picture, z represents the noise input to the G network, G(z) represents the picture generated by the G network, and D(·) represents the probability that the D network judges whether the picture is real.

[0082] Optionally, the perceptual loss may be the difference between the multi-dimensional feature values ​​of the color image and the line drawing colored image to be verified generated by the generator G, which are obtained by inputting the two into a VGG19 (Visual Geometry Group 19, a neural network structure with 19 hidden layers) network.

[0083] The above technical solution is based on the GAN network model, combined with brush stroke information, by introducing the pre-trained model Sketcha-a-Net and modifying the traditional U-Net network, thereby improving the network structure of the generated network and realizing automatic coloring of line draft images.

[0084] The technical solution of this embodiment obtains a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed, and inputs the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model, so as to generate a line draft colored image to be verified through the image generation module, and obtains an original sample image of the line draft image to be processed, and inputs the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module, thereby determining a loss value of the image processing model according to the image authenticity data and labeled data that matches the image authenticity data, and then training the image processing model according to the loss value, so as to solve the problem of checkerboard effect in the generated image caused by the existing image processing method, and can eliminate the checkerboard effect of the automatically generated line draft colored image, thereby improving the image quality of the automatically generated line draft colored image.

[0085] Embodiment 2

[0086] Figure 3 This is a flowchart of an image processing method provided by the second embodiment of the present invention. This embodiment is applicable to the case of eliminating the checkerboard effect of automatically generated line drawing coloring images. The method can be executed by an image processing device, which can be implemented by software and / or hardware, and can generally be directly integrated into an electronic device that executes the method. The electronic device can be a terminal device or a server device. The embodiment of the present invention does not limit the type of electronic device that executes the image processing method. Specifically, Figure 3 As shown, the image processing method may specifically include the following steps:

[0087] S310, obtaining a line drawing image to be processed.

[0088] S320, performing brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed; wherein the brush stroke image to be processed is a fused image of the line draft image to be processed and the pixel point image to be processed.

[0089] The stroke image to be processed may be an image obtained by performing stroke processing on the line draft image to be processed based on the line draft image to be processed. Specifically, the stroke image to be processed may be a fusion image of the line draft image to be processed and the pixel point image to be processed.

[0090] In the embodiment of the present invention, after obtaining the line draft image to be processed, the line draft image to be processed may be further subjected to brush stroke processing to obtain the brush stroke image to be processed. Optionally, the brush stroke processing may be performed manually or automatically on the line draft image to be processed, and the embodiment of the present invention is not limited thereto.

[0091] S330, inputting the brush stroke image to be processed into an image generation module of a trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generation module to obtain a target line drawing colored image.

[0092] The target line drawing colored image may be a color image obtained after image processing is performed on the stroke image to be processed.

[0093] In an embodiment of the present invention, after performing brush stroke processing on the line draft image to be processed to obtain the brush stroke image to be processed, the brush stroke image to be processed can be further input into an image generation module of a trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generation module, thereby obtaining a target line draft colored image.

[0094] S340 , performing noise filtering processing on the target line draft colored image by using a joint bilateral filter to obtain a target image.

[0095] The joint bilateral filter can be used to filter out residual noise of the target line drawing colored image. The target image can be an image obtained by performing noise filtering on the target line drawing colored image.

[0096] In an embodiment of the present invention, after the image generation module performs image processing on the brush stroke image to be processed to obtain the target line drawing colored image, the target line drawing colored image can be further subjected to noise filtering processing by a joint bilateral filter to obtain the target image.

[0097] The above technical solution can make the target image more vivid by using a joint bilateral filter to perform noise filtering on the target line drawing colored image.

[0098] In a specific example, Figure 4 is an example flow chart of an image processing method provided by Embodiment 2 of the present invention. Figure 4 As shown, the image processing method may specifically include the following contents:

[0099] The image obtained by brush stroke processing on the line draft image (that is, the brush stroke image to be processed) is input into the generator G of the successfully trained GAN network model; the generator G of the GAN network model automatically uses the color cues in the brush stroke image to be processed to color the line draft image at its corresponding position, and diffuses the color to the edge and stops; the residual noise in the colored image (that is, the target line draft colored image) is filtered out by a joint bilateral filter to obtain the final image (that is, the target image).

[0100] The above technical solution can make the generated line drawing colored image look more vivid by using a combined bilateral filter to filter out noise from the colored image.

[0101] The technical solution of this embodiment obtains a line draft image to be processed, performs brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed, and inputs the brush stroke image to be processed into an image generation module of a trained image processing model, performs image processing on the brush stroke image to be processed by the image generation module to obtain a target line draft colored image, and performs noise filtering processing on the target line draft colored image by a joint bilateral filter to obtain a target image, thereby solving the problem of checkerboard effect in the generated image caused by the existing image processing method, and can eliminate the checkerboard effect of the automatically generated line draft colored image, thereby improving the image quality of the automatically generated line draft colored image.

[0102] Embodiment 3

[0103] Figure 5 is a schematic diagram of an image processing model training device provided by Embodiment 3 of the present invention, such as Figure 5 As shown, the device includes: an image acquisition module 510, a line draft coloring image generation module 520 to be verified, an image authenticity data generation module 530 and an image processing model training module 540, wherein:

[0104] An image acquisition module 510 is used to acquire a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed;

[0105] The line draft coloring image generation module 520 to be verified is used to input the line draft image to be processed and the pixel point image to be processed into the image generation module of the image processing model, so as to generate the line draft coloring image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer;

[0106] An image authenticity data generating module 530 is used to obtain an original sample image of the line draft image to be processed, and input the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module;

[0107] The image processing model training module 540 is used to determine the loss value of the image processing model based on the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value.

[0108] The technical solution of this embodiment obtains a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed, and inputs the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model, so as to generate a line draft colored image to be verified through the image generation module, and obtains an original sample image of the line draft image to be processed, and inputs the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module, thereby determining a loss value of the image processing model according to the image authenticity data and labeled data that matches the image authenticity data, and then training the image processing model according to the loss value, so as to solve the problem of checkerboard effect in the generated image caused by the existing image processing method, and can eliminate the checkerboard effect of the automatically generated line draft colored image, thereby improving the image quality of the automatically generated line draft colored image.

[0109] Optionally, the line draft coloring image generation module 520 to be verified can be specifically used to: extract features of the line draft image to be processed through the Sketch-a-Net model to obtain the line draft image features to be processed of the line draft image to be processed; obtain the generated image features generated by the image generation module through the image generation module, and input the line draft image features to be processed into the image generation module; and perform feature splicing of the line draft image features to be processed and the generated image features through the image generation module.

[0110] Optionally, the line drawing coloring image generation module 520 to be verified can also be specifically used to connect the preset convolution layer and the preset deconvolution layer of the full convolution structure of the image generation module through a long connection method.

[0111] Optionally, the image acquisition module 510 can be specifically used to: acquire an original sample image; perform edge extraction on the original sample image using an edge extraction model to obtain a line drawing image to be processed; perform pixel extraction on the original sample image using a pixel extraction model to obtain a pixel image to be processed.

[0112] Optionally, the image processing model training module 540 can be specifically used to: determine the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value of the image processing model based on the image authenticity data and the labeled data matching the image authenticity data; determine the loss value through the adversarial loss value, L1 loss value, total variation loss value and perceptual loss value.

[0113] The above-mentioned image processing model training device can execute the image processing model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the image processing model training method provided by any embodiment of the present invention.

[0114] Since the image processing model training device introduced above is a device that can execute the image processing model training method in the embodiment of the present invention, based on the image processing model training method introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the image processing model training device of the present embodiment and its various variations, so how the image processing model training device implements the image processing model training method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device used by the image processing model training method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0115] Embodiment 4

[0116] Figure 6 is a schematic diagram of an image processing device provided by Embodiment 4 of the present invention, such as Figure 6 As shown, the device includes: a to-be-processed line draft image acquisition module 610, a to-be-processed stroke image acquisition module 620, a target line draft coloring image generation module 630 and a target image generation module 640, wherein:

[0117] The to-be-processed line draft image acquisition module 610 is used to acquire the to-be-processed line draft image;

[0118] The brushstroke image acquisition module 620 is used to perform brushstroke processing on the line draft image to be processed to obtain a brushstroke image to be processed; the brushstroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed;

[0119] A target line draft coloring image generating module 630 is used to input the brush stroke image to be processed into an image generating module of a trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generating module to obtain a target line draft coloring image;

[0120] The target image generation module is used to perform noise filtering processing on the target line draft colored image through a joint bilateral filter to obtain a target image.

[0121] The technical solution of this embodiment obtains a line draft image to be processed, performs brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed, and inputs the brush stroke image to be processed into an image generation module of a trained image processing model, performs image processing on the brush stroke image to be processed by the image generation module to obtain a target line draft colored image, and performs noise filtering processing on the target line draft colored image by a joint bilateral filter to obtain a target image, thereby solving the problem of checkerboard effect in the generated image caused by the existing image processing method, and can eliminate the checkerboard effect of the automatically generated line draft colored image, thereby improving the image quality of the automatically generated line draft colored image.

[0122] The above-mentioned image processing device can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the image processing method provided by any embodiment of the present invention.

[0123] Since the image processing device described above is a device that can execute the image processing method in the embodiment of the present invention, based on the image processing method described in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the image processing device in the embodiment and its various variations, so how the image processing device implements the image processing method in the embodiment of the present invention is not described in detail here. As long as a person skilled in the art implements the device used by the image processing method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0124] Embodiment 5

[0125] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. Figure 7 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0126] like Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, a memory 28, and a bus 18 that connects various system components (including the memory 28 and the processor 16).

[0127] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0128] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0129] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0130] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0131] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0132] The processor 16 executes various functional applications and data processing by running the program stored in the memory 28, thereby realizing the image processing model training method provided in the first embodiment of the present invention: obtaining a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed; inputting the line draft image to be processed and the pixel point image to be processed into the image generation module of the image processing model, so as to generate a line draft colored image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer; obtaining the original sample image of the line draft image to be processed, and inputting the original sample image and the line draft colored image to be verified into the image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module; determining the loss value of the image processing model according to the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value.

[0133] Or the processor 16 executes various functional applications and data processing by running the program stored in the memory 28 to implement the image processing method provided in the second embodiment of the present invention: obtaining the line draft image to be processed; performing brush stroke processing on the line draft image to be processed to obtain the brush stroke image to be processed; wherein the brush stroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed; inputting the brush stroke image to be processed into the image generation module of the trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generation module to obtain a target line draft colored image; performing noise filtering processing on the target line draft colored image through a joint bilateral filter to obtain a target image.

[0134] Embodiment 6

[0135] Embodiment 6 of the present invention further provides a computer storage medium storing a computer program, which, when executed by a computer processor, is used to execute the image processing model training method described in the above embodiment 1 of the present invention: obtaining a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed; inputting the line draft image to be processed and the pixel point image to be processed into an image generation module of the image processing model, so as to generate a line draft colored image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer; obtaining an original sample image of the line draft image to be processed, and inputting the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module; determining a loss value of the image processing model according to the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value.

[0136] Or the computer program is used to execute the image processing method described in the above-mentioned embodiment 2 of the present invention when executed by a computer processor: obtaining a line draft image to be processed; performing brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed; wherein the brush stroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed; inputting the brush stroke image to be processed into an image generation module of a trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generation module to obtain a target line draft colored image; performing noise filtering processing on the target line draft colored image through a joint bilateral filter to obtain a target image.

[0137] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0138] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0140] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for training an image processing model, characterized in that: include: Acquire a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed; Inputting the line draft image to be processed and the pixel point image to be processed into an image generation module of an image processing model, so as to generate a line draft colored image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer; Acquire an original sample image of the line draft image to be processed, and input the original sample image and the line draft colored image to be verified into an image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module; Determining a loss value of the image processing model according to the image authenticity data, the labeled data matched by the image authenticity data, the original sample image, and the line draft colored image to be verified, so as to train the image processing model according to the loss value; Before the image generation module generates the line drawing colored image to be verified, the method further includes: Performing feature extraction on the line draft image to be processed by using the Sketch-a-Net model to obtain the line draft image features to be processed of the line draft image to be processed; Acquire the generated image features generated by the image generation module through the image generation module, and input the line draft image features to be processed into the image generation module; Performing feature splicing of the to-be-processed line draft image features and the generated image features through the image generation module; The generated image features are features in the line draft image to be processed obtained by the image generation module.

2. The method according to claim 1, characterized in that Before the image generation module generates the line drawing colored image to be verified, the method further includes: The preset convolution layer and the preset deconvolution layer of the full convolution structure of the image generation module are connected through a long connection method.

3. The method according to claim 1, characterized in that: Before acquiring the line draft image to be processed and the pixel point image to be processed matching the line draft image to be processed, the method further includes: Get the original sample image; Performing edge extraction on the original sample image using an edge extraction model to obtain the line draft image to be processed; Pixel extraction is performed on the original sample image using a pixel extraction model to obtain the pixel image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the loss value of the image processing model according to the image authenticity data, the labeled data matched by the image authenticity data, the original sample image and the line draft colored image to be verified includes: determining an adversarial loss value of the image processing model based on the image authenticity data and the labeled data matching the image authenticity data; Determine the L1 loss value and the perceptual loss value of the image processing model according to the original sample image and the line draft colored image to be verified; Determine the total variation loss value of the image processing model according to the line drawing colored image to be verified; The loss value is determined by the adversarial loss value, the L1 loss value, the perceptual loss value and the total variation loss value.

5. An image processing method, characterized in that: include: Get the line drawing image to be processed; Performing brush stroke processing on the line draft image to be processed to obtain a brush stroke image to be processed; wherein the brush stroke image to be processed is a fused image of the line draft image to be processed and the pixel point image to be processed; Inputting the brushstroke image to be processed into an image generation module of a trained image processing model, so as to perform image processing on the brushstroke image to be processed through the image generation module to obtain a target line drawing colored image; the trained image processing model is an image processing model based on deep learning; The target line drawing colored image is subjected to noise filtering processing by using a joint bilateral filter to obtain a target image.

6. An image processing model training device, characterized in that: include: An image acquisition module, used to acquire a line draft image to be processed and a pixel point image to be processed that matches the line draft image to be processed; A line draft coloring image generation module to be verified, used for inputting the line draft image to be processed and the pixel point image to be processed into the image generation module of the image processing model, so as to generate the line draft coloring image to be verified through the image generation module; wherein the image generation module includes a full convolution structure, and the target deconvolution layer of the full convolution structure includes an upsampling layer and a convolution layer; An image authenticity data generation module, used for obtaining an original sample image of the line draft image to be processed, and inputting the original sample image and the line draft colored image to be verified into the image discrimination module of the image processing model, so as to generate image authenticity data through the image discrimination module; An image processing model training module, used to determine a loss value of the image processing model according to the image authenticity data and the labeled data matching the image authenticity data, so as to train the image processing model according to the loss value; The module for generating colored line draft images to be verified is further used for performing feature extraction on the line draft images to be processed through the Sketch-a-Net model to obtain the line draft image features to be processed of the line draft images to be processed; obtaining the generated image features generated by the image generation module through the image generation module, and inputting the line draft image features to be processed into the image generation module; performing feature splicing on the line draft image features to be processed and the generated image features through the image generation module; wherein the generated image features are features in the line draft image to be processed obtained through the image generation module.

7. An image processing device, characterized in that: include: A module for acquiring line draft images to be processed, used for acquiring line draft images to be processed; A brushstroke image acquisition module to be processed is used to perform brushstroke processing on the line draft image to be processed to obtain a brushstroke image to be processed; the brushstroke image to be processed is a fusion image of the line draft image to be processed and the pixel point image to be processed; a target line draft coloring image generating module, used for inputting the brush stroke image to be processed into the image generating module of the trained image processing model, so as to perform image processing on the brush stroke image to be processed through the image generating module, and obtain a target line draft coloring image; the trained image processing model is an image processing model based on deep learning; The target image generation module is used to perform noise filtering processing on the target line draft colored image through a joint bilateral filter to obtain a target image.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing model training method as described in any one of claims 1 to 4, or implement the image processing method as described in claim 5.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the image processing model training method as described in any one of claims 1 to 4, or implements the image processing method as described in claim 5.

Images