Method and device for converting image into sand painting based on deep learning

Abstract

The invention discloses a deep learning-based image-to-sand painting method and device, and the method comprises the steps: constructing a style picture and a content picture into a new data set class, carrying out the packaging, and completing the picture loading processing; preprocessing the loaded style picture and content picture; constructing a convolutional neural network; performing content processing on the preprocessed style picture and content picture through a convolutional neural network, and initializing to generate a target image; extracting style features and content features; performing iterative training on the convolutional neural network through loss function calculation; and processing the target image generated by initialization through the trained convolutional neural network to obtain a processed target image, performing post-processing, and outputting the post-processed target image to obtain a final target image. The content loss can be controlled, the style loss can be reduced, the optical color characteristics of the image can be kept, and the quality of the generated image can be evaluated through clear indexes.

Description

Technical Field

[0001] The present invention relates to the technical field of image conversion, and in particular to a method and device for converting an image into a sand painting based on deep learning. Background Art

[0002] The existing technical solution uses an image style transfer idea based on the VGG-19 convolutional neural network, that is, through an image style transfer algorithm based on deep learning, the algorithm can combine an input image as a content image with another style image as a style image to generate a new "stitched image". In this algorithm, for the VGG network, the convolution layer closer to the input layer is more biased towards the local style features of the style image for style extraction, and the convolution layer closer to the output layer is better at overall control of the style; in terms of content extraction, the convolution layer close to the input layer has a better effect on style restoration, while the convolution layer close to the output layer has an unsatisfactory effect on the restoration of the convolution layer, with more noise points in the picture, and will cause certain deformation and collapse of the picture. Therefore, it is necessary to intercept the output of the neural network close to the input layer for content loss calculation, and intercept the content of the neural network close to the input and output layers for style loss calculation.

[0003] At present, the existing technical solutions have the following deficiencies:

[0004] First, the coefficients of the content loss function and the style loss function are difficult to determine.

[0005] In the existing technical solution, the idea of ​​generation is to initially generate a completely noisy image, and then gradually reduce the various losses of the generated image to the style image and the content image as the iterations increase during training, so that the image is gradually restored to a mixed image with both style features of the style image and content features of the content image. In this case, the content loss and style loss of the generated image gradually decrease with the increase in the number of iterations, which is also a general method in the field of image generation style transfer. The disadvantage of this idea is that it is difficult to balance the degree of content and style conversion in the image generated by this method, and it is extremely difficult to set the parameters. Although painting with sand and light boxes is collectively referred to as sand painting, the painting style of each painter varies greatly. For sand paintings of different painting styles, it is necessary to balance their content loss coefficients and style loss coefficients separately, and it is also necessary to re-determine the model parameters. The determination of such parameters must be determined in advance during the training process, which is undoubtedly difficult to set accurately.

[0006] Second, the problem of inconsistent color of generated images cannot be completely overcome.

[0007] This defect is not caused by insufficient training times or poor model debugging. In the final analysis, it is because the loss function of the neural network of this system (whether it is the content loss function or the style loss function) cannot completely separate color from content and style. This is also a defect of the current loss function algorithm itself.

[0008] Third, there is no indicator to accurately measure the quality of generated sand paintings.

[0009] The evaluation of sand painting is ultimately an evaluation of its artistry, but the evaluation of artistry varies from person to person. Some people think this kind of generation effect is good, while others think that kind of generation is good. Perhaps judging the artistry of a sand painting is a matter of opinion, but there is still a lack of clear indicators on how to exclude sand paintings that are not well generated.

[0010] Fourth, there is no complete user interaction system platform.

[0011] At present, there are precedents for user platforms that convert images into oil paintings and comics based on deep learning (such as Diffusion and Novel Ai), but there is no user online interactive platform for converting images into sand paintings, which is also a shortcoming of the current technical solutions.

[0012] Therefore, how to invent a method that can control content loss and reduce style loss while retaining the optical color characteristics of the image and evaluating the quality of the generated image through clear indicators has become an urgent problem to be solved. Summary of the invention

[0013] To this end, the present invention provides a deep learning-based image-to-sand painting method and device, which can control content loss and reduce style loss while retaining the optical color characteristics of the image and evaluating the quality of the generated image through clear indicators.

[0014] In order to achieve the above object, the present invention provides the following technical solution: a method for converting an image into a sand painting based on deep learning, comprising:

[0015] The style image and the content image are constructed into a new data set class by setting the data set class, and the new data set class is encapsulated by setting the function to complete the image loading process;

[0016] Preprocessing the loaded style picture and the content picture according to a set method to obtain the preprocessed style picture and the content picture;

[0017] Constructing a convolutional neural network, performing content processing on the preprocessed style image and the content image through the convolutional neural network, and initializing and generating a target image;

[0018] Extracting style features and content features from the preprocessed style image and content image respectively;

[0019] According to the extracted style features and content features, the convolutional neural network is iteratively trained by loss function calculation to obtain the trained convolutional neural network; the initialized target image is processed by the trained convolutional neural network to obtain a processed target image;

[0020] Post-processing the processed target image so that the processed target image retains the optical color characteristics of the content image;

[0021] The target image after the post-processing is output to obtain a final target image.

[0022] As a preferred solution for the image-to-sand painting method based on deep learning, in the process of constructing the style image and the content image into a new dataset class by setting the dataset class, and encapsulating the new dataset class by setting the function, the PyTorch dataset class Dataset is used as the parent class to construct a new dataset class DataSetOri, and the new dataset class is encapsulated by overriding the init constructor, getitem function and len function.

[0023] As a preferred scheme of the image-to-sand painting method based on deep learning, in the process of pre-processing the loaded style picture and the content picture according to a set method, the setting method includes trimming the size of the style picture and the content picture and standardizing and normalizing each pixel of the style picture and the content picture.

[0024] As a preferred solution of the image-to-sand painting method based on deep learning, in the process of performing content processing on the pre-processed style image and the content image through the convolutional neural network and initializing the generation of the target image, the content of the content image is copied to the generated target image; through the iterative training processing of the convolutional neural network, the target image is transformed from the content image into a content and style mixed image.

[0025] As a preferred solution of the image-to-sand painting method based on deep learning, in the process of iteratively training the convolutional neural network according to the extracted style features and the content features through loss function calculation, the steps of iteratively training the convolutional neural network are:

[0026] Inputting the extracted style features and content features into the convolutional neural network;

[0027] Calculate content loss and style loss through loss function;

[0028] Obtaining the total loss of each round of the convolutional neural network training according to the obtained content loss and the style loss;

[0029] Calculate the impact of each parameter in the convolutional neural network on the total loss through back propagation;

[0030] The parameter weights in the convolutional neural network are optimized and updated according to the influence of each parameter in the convolutional neural network on the total loss.

[0031] As a preferred solution of the image-to-sand painting method based on deep learning, in the process of iteratively training the convolutional neural network according to the extracted style features and content features, the content loss is calculated by using the square sum of differences strategy; the style loss is calculated by using the Gram matrix strategy;

[0032] The calculation formula of the content loss is:

[0033]

[0034] Where, L conetnt is the content loss; F G and F I are content pictures and style pictures, and the superscript l is the specified color channel;

[0035] The calculation formula of the style loss is:

[0036]

[0037] In the formula, G [l] (k 1 , k 2 ) is the Gram matrix; G S [l] (k 1 , k 2 ) and G G [l] (k 1 , k 2 ) are the Gram matrices of the two pictures respectively; k 1 and k 2 The matrix G [l] Two channels in ; and k 1 and k 2 Neurons at the same location in the channel.

[0038] As a preferred solution of the image-to-sand painting method based on deep learning, in the process of post-processing the processed target image, the RGB mean and standard deviation are calculated by the calRGBMean function and the calRGBStd function; the RGB mean and standard deviation are tensorized, multiplied and added with each pixel of the processed target image, so as to complete the post-processing of the processed target image.

[0039] The present invention also provides a deep learning-based image-to-sand painting processing device, based on the above deep learning-based image-to-sand painting method, comprising:

[0040] The image loading module is used to construct the style image and the content image into a new data set class by setting the data set class, and encapsulate the new data set class by setting the function to complete the image loading process;

[0041] An image preprocessing module, used to preprocess the loaded style image and the content image according to a set method to obtain the preprocessed style image and the content image;

[0042] Convolutional neural network building module, used to build convolutional neural networks;

[0043] A target image initialization generation module, used to perform content processing on the preprocessed style image and the content image through the convolutional neural network to initialize and generate a target image;

[0044] A picture feature extraction module, used to extract style features and content features from the preprocessed style picture and content picture respectively;

[0045] A convolutional neural network training and processing module, configured to iteratively train the convolutional neural network according to the extracted style features and content features through loss function calculation to obtain the trained convolutional neural network; and process the initialized target image through the trained convolutional neural network to obtain a processed target image;

[0046] An image post-processing module, used for post-processing the processed target image so that the processed target image retains the optical color characteristics of the content image;

[0047] The image output module is used to output the target image after the post-processing to obtain the final target image.

[0048] As a preferred solution for an image-to-sand painting processing device based on deep learning, in the picture loading module, in the process of constructing the style picture and the content picture into a new dataset class by setting the dataset class, and encapsulating the new dataset class by setting the function, the PyTorch dataset class Dataset is used as the parent class to construct a new dataset class DataSetOri, and the new dataset class is encapsulated by overriding the init constructor, getitem function and len function.

[0049] As a preferred solution for an image-to-sand painting processing device based on deep learning, in the image preprocessing module, in the process of preprocessing the loaded style image and the content image according to a set method, the set method includes trimming the size of the style image and the content image and standardizing and normalizing each pixel of the style image and the content image.

[0050] As a preferred solution for the image-to-sand painting processing device based on deep learning, in the target image initialization generation module, while performing content processing on the pre-processed style image and the content image through the convolutional neural network and initializing the generation of the target image, the content of the content image is copied to the generated target image; through the iterative training processing of the convolutional neural network, the target image is transformed from the content image into a content and style mixed image.

[0051] As a preferred solution of the image-to-sand painting processing device based on deep learning, the convolutional neural network training and processing module includes:

[0052] An extraction feature input submodule, used for inputting the extracted style features and the content features into the convolutional neural network;

[0053] The loss calculation submodule is used to calculate the content loss and style loss through the loss function;

[0054] A total loss acquisition submodule, used to obtain the total loss of each round of the convolutional neural network training according to the obtained content loss and style loss;

[0055] A back propagation calculation submodule, used to calculate the influence of each parameter in the convolutional neural network on the total loss through back propagation;

[0056] The parameter weight optimization submodule is used to optimize and update the parameter weights in the convolutional neural network according to the influence of each parameter in the convolutional neural network on the total loss.

[0057] As a preferred solution of the image-to-sand painting processing device based on deep learning, in the convolutional neural network training and processing module, in the process of iteratively training the convolutional neural network according to the extracted style features and content features, through loss function calculation, the content loss is calculated by using the difference square sum strategy; the style loss is calculated by using the Gram matrix strategy;

[0058] The calculation formula of the content loss is:

[0059]

[0060] Where, L conetnt is the content loss; F G and F I are content pictures and style pictures, and the superscript l is the specified color channel;

[0061] The calculation formula of the style loss is:

[0062]

[0063] In the formula, G [l] (k 1 , k 2 ) is the Gram matrix; G S [l] (k 1 , k 2 ) and G G [l] (k 1 , k 2 ) are the Gram matrices of the two pictures respectively; k 1 and k 2 The matrix G [l] Two channels in ; and k 1 and k 2 Neurons at the same location in the channel.

[0064] As a preferred solution of the image-to-sand painting processing device based on deep learning, in the image post-processing module, in the process of post-processing the processed target image, the RGB mean value and standard deviation are calculated by the calRGBMean function and the calRGBStd function; the RGB mean value and standard deviation are tensorized, multiplied and added with each pixel of the processed target image, to complete the post-processing of the processed target image.

[0065] The present invention has the following advantages: constructing a style picture and a content picture into a new data set class by setting a data set class, and encapsulating the new data set class by setting a function to complete the picture loading process; preprocessing the loaded style picture and the content picture in a set manner to obtain the preprocessed style picture and the content picture; constructing a convolutional neural network; performing content processing on the preprocessed style picture and the content picture by the convolutional neural network to initialize and generate a target image; extracting style features and content features from the preprocessed style picture and the content picture respectively; iteratively training the convolutional neural network according to the extracted style features and content features through loss function calculation to obtain the trained convolutional neural network; processing the initialized target image through the trained convolutional neural network to obtain a processed target image; post-processing the processed target image so that the processed target image retains the optical color features of the content picture; outputting the post-processed target image to obtain a final target image. The present invention solves the problem that the coefficients of the content loss function and the style loss function in the prior art are difficult to determine by using the content image as the generated image, then controlling the content loss and reducing the style loss; introduces and improves the "multiply" solution, first decolorizes the imperfect generated image, and then uses multiply to recolor it, while retaining the optical color characteristics of the image, solving the problem of inconsistent color of the generated image in the prior art; introduces the new concept of "noise cluster" to judge the roughness of the generated image. The more noise clusters there are (or connected into pieces), the worse the picture of the generated image is, and the generation parameter combination should be rejected. The problem that there is no indicator to accurately measure the quality of the sand painting generated by the prior art is solved. At the same time, the entire algorithm of the present invention is encapsulated in a Python class, and the class is mounted on the HTTP request response function of Django to realize the server application. The problem that the prior art does not have a perfect user interaction system platform is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0067] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0068] Figure 1 A schematic diagram of a process of converting an image into a sand painting based on deep learning provided in Example 1 of the present invention;

[0069] Figure 2 A user system timing diagram in a method for converting an image to a sand painting based on deep learning provided in Example 1 of the present invention;

[0070] Figure 3 A business logic flow chart of a method for converting an image to a sand painting based on deep learning provided in Example 1 of the present invention;

[0071] Figure 4 A schematic diagram of a neural network layer for calculating content loss and style loss in a deep learning-based image-to-sand painting method provided in Example 1 of the present invention;

[0072] Figure 5 A schematic diagram of the influence of the weight of the style loss function on the generated image in a method for converting an image to a sand painting based on deep learning provided in Example 1 of the present invention;

[0073] Figure 6 A schematic diagram showing the effect of the number of iterations on the generated image in a deep learning-based image-to-sand painting method provided in Example 1 of the present invention;

[0074] Figure 7 A schematic diagram showing the effect of the number of iterations on the generated image under the parameter 1e5 in a deep learning-based image-to-sand painting method provided in Example 1 of the present invention;

[0075] Figure 8 A schematic diagram of noise clusters in a deep learning-based image-to-sand painting method provided in Example 1 of the present invention;

[0076] Fig. 9 A schematic diagram of a simple system architecture built according to the present invention in a method for converting an image to a sand painting based on deep learning provided in Example 1 of the present invention;

[0077] Fig.10 This is a schematic diagram of the architecture of an image-to-sand painting processing device based on deep learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0078] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Example 1

[0080] See also Figure 1 , Figure 2 and Figure 3 Embodiment 1 of the present invention provides a method for converting an image into a sand painting based on deep learning, comprising the following steps:

[0081] S1. Constructing the style image and the content image into a new data set class by setting the data set class, and encapsulating the new data set class by setting the function to complete the image loading process;

[0082] S2, preprocessing the loaded style picture and the content picture according to a set method to obtain the preprocessed style picture and the content picture;

[0083] S3, constructing a convolutional neural network, wherein the convolutional neural network is used to complete the style transformation of the image;

[0084] S4, performing content processing on the preprocessed style image and the content image through the convolutional neural network to initialize and generate a target image;

[0085] S5, extracting style features and content features from the preprocessed style image and content image respectively;

[0086] S6, according to the extracted style features and the content features, the convolutional neural network is iteratively trained by loss function calculation to obtain the trained convolutional neural network; the initialized target image is processed by the trained convolutional neural network to obtain a processed target image;

[0087] S7, post-processing the processed target image so that the processed target image retains the optical color characteristics of the content image;

[0088] S8. Output the post-processed target image to obtain a final target image.

[0089] See also Figure 2 , the present invention does not involve the database structure, so the timing Figure 3The three levels are user (operating front end), back-end service program (receiving front-end requests) and neural network training (business logic layer). The three modules (or three sequential processes) have a strict time and space structure and are arranged in sequence in the figure.

[0090] Among them, the "neural network training" part receives a content image in a three-channel RGB mode and a style image in a three-channel RGB mode, and finally synthesizes a synthetic image in RGB mode, which has both the content features of the content image and the style features of the style image.

[0091] In this embodiment, in step S1, the style image and the content image are constructed into a new data set class by setting the data set class, and the new data set class is encapsulated by setting the function to complete the image loading process;

[0092] Specifically, the PyTorch dataset class Dataset is used as the parent class to construct a new dataset class DataSetOri, and the new dataset class is encapsulated by rewriting the init constructor, getitem function, and len function.

[0093] The getitem function is a function required to extract a single element in a container. The Image.open() function in PIL Image is called in the function to interact with the operating system file system. Outside the class, the technical solution initializes the data set with the paths of the style image and the content image, and then uses the getitem function to find the content image and the style image itself.

[0094] In this embodiment, in step S2, the loaded style picture and the content picture are preprocessed according to a set method to obtain the preprocessed style picture and the content picture;

[0095] Specifically, since it has been agreed that the channel of the image is in RGB mode, the image preprocessing is mainly divided into two parts. The first is to trim the image size, and the second is to standardize and normalize each pixel of the image.

[0096] For the standardization of image pixels, the value of each channel of each pixel is subtracted from its own average value and divided by its variance. Due to the good compatibility between PyTorch and Numpy, Numpy is used here to array the image, and the average and standard deviation functions provided by Numpy are used to calculate the entire image. In the custom standardization function imagePreProcess, the technical solution uses the transforms class in PyTorch to implement all the processing at one time, namely "size cropping", "converting to tensor", "standardization", and finally returns the processed image.

[0097] In this embodiment, in step S3, a convolutional neural network is constructed, and the style transformation of the image is completed through the convolutional neural network.

[0098] The structure of the convolutional neural network can be replaced and improved. The purpose of the convolutional neural network is to extract graphic features using the size of the convolution kernel. Therefore, if you do not use a neural network model for targeted training, but use a pre-trained neural network, it will also have a certain effect. And the structure of the network can be adjusted, such as introducing a residual network.

[0099] In this embodiment, in step S4, the pre-processed style image and the content image are subjected to content processing by the convolutional neural network to initialize and generate a target image;

[0100] Specifically, when initializing the generated image, a noise image is not generated, but the content of the content image is copied into the generated image. In this way, the iterative process in training becomes a pure style transfer process, from the content image to a mixed image of content and style. In this case, the content loss of the image increases from zero as the training progresses, while the style loss gradually decreases.

[0101] In this embodiment, in step S5, style features and content features are respectively extracted from the pre-processed style picture and content picture.

[0102] In this embodiment, in step S6, the convolutional neural network is iteratively trained according to the extracted style features and the content features through loss function calculation to obtain the trained convolutional neural network; the initialized target image is processed by the trained convolutional neural network to obtain a processed target image;

[0103] Specifically, the steps of iterative training of the convolutional neural network are:

[0104] S61, inputting the extracted style features and content features into the convolutional neural network;

[0105] S62, calculating the content loss and the style loss through the loss function;

[0106] For details, see Figure 4 , the content loss is calculated by the sum of squared differences strategy; the style loss is calculated by the Gram matrix strategy;

[0107] The classic algorithm for calculating content loss is to calculate the sum of squared differences. You only need to compare the difference between each pixel in each channel of the content image and the generated image in turn, and then add them together to get the total content loss. For the ideal case (that is, the content of the two images is exactly the same), the difference between each pixel in each channel is 0, so the sum is also 0 - this is also the goal of reducing content loss.

[0108] The calculation formula of the content loss is:

[0109]

[0110] Where, L conetnt is the content loss; F G and F I are content images and style images. The superscript l indicates the specified color channel. The outermost coefficient 1 / 2 is the normalization coefficient, which is optional in actual program vision.

[0111] The method used in the present invention to calculate the style loss is the correlation matrix method. Since Gram is often used to represent the correlation matrix in linear algebra, it is also called the Gram matrix method. Among them, the style content loss needs to examine the different pixel differences between the same color channels in different pictures, and the calculation of style loss needs to examine the correlation between different color channels in the same picture. The adjacency matrix method is most commonly used in computers. For a neural network, the value in each neuron of a certain layer can be compared to a pixel value, and each layer in the neural network also has several neuron channels, which are similar to color channels.

[0112] So suppose a neural network layer has n c channels, then this matrix should also be n c ×n c A square matrix of size: let this matrix be called matrix G [l] (where l is the network layer). And we want to explore two channels k 1 and k 2 When there is a relationship between , we can directly search through the square matrix, denoted by G [l] (k 1 , k 2 ). As for the specific calculation of this value, we only need to traverse channel k 1and channel k 2 For all neurons in the same position, we can just multiply them (or calculate their covariance and correlation coefficient).

[0113]

[0114] In the formula, and k 1 and k 2 Neurons at the same location in the channel.

[0115] The above formula is the calculation method of Gram matrix. With the support of array calculation of graphics card, this calculation is very fast. Then the style loss should be the difference of Gram matrix of two pictures and then add the absolute value of determinant, as shown in the following formula:

[0116]

[0117] In the formula, G S [l] (k 1 , k 2 ) and G G [l] (k 1 , k 2 ) are the Gram matrices of the two pictures respectively.

[0118] S63. Obtain the total loss of each round of the convolutional neural network training according to the obtained content loss and the style loss;

[0119] S64, calculating the influence of each parameter in the convolutional neural network on the total loss by back propagation;

[0120] Specifically, after calculating the loss function, it is necessary to reduce this loss in each round of training so that the generated image is close to the content image in terms of content and closer to the style image in terms of style. This process is completed in back propagation. Back propagation calculates the impact of each parameter on the loss by propagating the gradient of the loss function calculation result from the output layer of the neural network to the input layer of the network.

[0121] S65. Optimize and update the parameter weights in the convolutional neural network according to the influence of each parameter in the convolutional neural network on the total loss.

[0122] Specifically, this back-propagation process can use the chain rule to calculate the gradient of each layer, and then effectively update and optimize the network parameters. This method of reducing losses can use the gradient descent algorithm. That is, in gradient descent, the algorithm updates the parameters along the negative gradient direction of the loss function to reduce the value of the loss function. Specifically, the update rule of gradient descent can be expressed as:

[0123]

[0124] Where θ is the parameter to be updated, α is called the learning rate (as a global variable of the system), is the gradient of the loss function with respect to θ.

[0125] When training a neural network, the gradient of the loss function with respect to the network parameters is calculated through back propagation, and then the gradient descent algorithm is used to update the parameters in the network. The frequency and amplitude of parameter updates are controlled by the learning rate α. A smaller learning rate may lead to slower convergence, but may be more stable; a larger learning rate may lead to an unstable training process or failure to converge. Therefore, in the selection of the learning rate, the learning rate is often set to a smaller value at the beginning, and then a larger α is continuously tried in multiple experiments until a situation of failure to converge occurs, and the largest α value that can converge is used as the learning rate.

[0126] In the actual process of sand painting transformation, the effect of "sand roughness" is analyzed first.

[0127] When other parameters are the same, the greater the weight of the style loss function, the greater the proportion of restoration and generation according to the style during model training. In other words, the richer the sand painting texture of the generated image, the more noise there will be, which is reflected in the user's vision as relatively rough gravel.

[0128] On the contrary, the smaller the weight of the style loss function, the smaller the proportion of restoration and generation according to the style during model training. In other words, the generated image will have less sand painting-like texture, and correspondingly, less noise (and the noise reduction process will be easier) - reflected in the user's vision, it will appear that the sand painting is very smooth and the gravel is very smooth.

[0129] First, we iterate the same image, only keeping the weight of the style loss function different and keeping other parameters the same, and we can get the following results:

[0130] like Figure 5 As shown, the three pictures represent the style loss function weights of the image are 1×10 3 , 1×10 4 , 1×10 5 The final result of the image is generated when the number of training iterations is 100. In the result of 1e3-100, the image is quite smooth with very little noise (as can be seen from the building on the right), but the texture of the sand painting (as can be seen from the tree trunks by the lake) is not very obvious, which is exactly the same as the previous analysis.

[0131] Next, evaluate the impact of the "number of iterations" to determine the appropriate number of iterations.

[0132] During the iteration process, the content loss will start from 0, rise to a small value and then fluctuate steadily - maintaining a certain degree of content restoration, but leaving possible space for style transfer; the style loss will continue to shrink from a large value at the beginning, indicating that stylization is progressing steadily; the noise loss will increase from the beginning and then slowly decrease, which corresponds to the training idea of ​​this system to control noise points from the later stage of training.

[0133] like Figure 6 As shown in the figure, the number of iterations has an immediate effect on the final effect of the generated image, which is needless to say. The impact of the number of iterations on the generated image effect can be seen from multiple angles such as the teaching building on the right, the trees, and the stones in the foreground. However, as the number of iterations increases, it is not difficult to find that the number of noise points in the generated image is excessively suppressed, especially in the process of 200 to 350 rounds, the unique granular style of sand painting is swept away, which shows that the gradual development of total variation denoising will lead to a "smooth overfitting" effect in the generated image.

[0134] Next, the test will change a set of parameters and modify the weight of the image style loss function to 1×10 5 , to observe how the increase in the number of iterations will affect the generated image, such as Figure 7 shown.

[0135] In order to solve the problem that there is no evaluation index for the quality of sand painting in the prior art, the present invention introduces the concept of "noise cluster", which reflects the overall quality of the image by identifying the concentration of noise points in the image, thereby excluding images with poor quality, such as Figure 8 According to the experimental results and by counting the number of "noise clusters" and the degree of image smoothness, the recommended setting ranges of the above two parameters can be given.

[0136] Please note that this does not mean that the parameter combinations in the recommended range will definitely generate excellent images, but that the image effects outside the range will definitely be poor.

[0137] Parameter combination <![CDATA[1×10 3 ]]> <![CDATA[1×10 4 ]]> <![CDATA[1×10 5 ]]> 50 rounds 10 Appear in pieces Appear in pieces 100 rounds 7 13 Appear in pieces 200 rounds 2 9 Appear in pieces 350 rounds 1 6 18

[0138] Table 1 Number of “noise clusters”

[0139] Parameter combination <![CDATA[1×10 3 ]]> <![CDATA[1×10 4 ]]> <![CDATA[1×10 5 ]]> 50 rounds Fine texture Fine texture Fine texture 100 rounds Fine texture Fine texture Fine texture 200 rounds Fine texture Fine texture Rough texture 350 rounds Completely textureless Rough texture Rough texture

[0140] Table 2 Image smoothness

[0141] According to the basic information in Table 1, we exclude the cases where “noise clusters” appear in patches and the cases where “noise clusters” are too many, and only (1×10 3, 100 rounds), (1×10 3 , 200 rounds), (1×10 3 , 350 rounds), (1×10 4 , 200 rounds) and (1×10 4 , 350 rounds) five parameter combinations have good effects.

[0142] According to the basic information in Table 2, excluding (1×10 3 , 350 rounds) and (1×10 4 , 350 rounds) because their generated images either completely lost the texture of the sand painting, or the texture was too rough and the picture looked blurred (this type of denoising has a limited effect on image restoration).

[0143] Therefore, the recommended parameter combination is only (1×10 3 , 100 rounds), (1×10 3 , 200 rounds) and (1×10 4 , 200 rounds) in three groups.

[0144] In this embodiment, in step S7, the processed target image is post-processed so that the processed target image retains the optical color characteristics of the content image;

[0145] Specifically, the processed target image is post-processed. This process is the inverse of the process of image standardization and normalization. The RGB mean and standard deviation obtained by the calRGBMean function and the calRGBStd function can be tensorized, and then multiplied and added with each pixel of the output image to restore the original color. In order to be on the safe side (the channel value is lower than 0 or higher than 255), the restored values ​​of the three RGB channels must be "cropped". After the cropping process, it is necessary to use the previous method of overlay to unify the colors.

[0146] In this embodiment, in step S8, the target image after the post-processing is output to obtain a final target image.

[0147] See also Fig. 9 , encapsulate the entire algorithm of the present invention in a Python class, mount the class on the HTTP request response function of Django, and then realize the server application.

[0148] In summary, the present invention constructs a style picture and a content picture into a new data set class by setting a data set class, and encapsulates the new data set class by setting a function to complete the picture loading process; preprocesses the loaded style picture and the content picture in a set manner to obtain the preprocessed style picture and the content picture; constructs a convolutional neural network; performs content processing on the preprocessed style picture and the content picture by the convolutional neural network to initialize and generate a target image; extracts style features and content features from the preprocessed style picture and the content picture respectively; according to the extracted style features and the content features, the convolutional neural network is iteratively trained through loss function calculation to obtain the trained convolutional neural network; processes the initialized target image through the trained convolutional neural network to obtain a processed target image; post-processes the processed target image so that the processed target image retains the optical color features of the content picture; outputs the post-processed target image to obtain a final target image. The present invention solves the problem that the coefficients of the content loss function and the style loss function in the prior art are difficult to determine by using the content image as the generated image, then controlling the content loss and reducing the style loss; introduces and improves the "multiply" solution, first decolorizes the imperfect generated image, and then uses multiply to recolor it, while retaining the optical color characteristics of the image, solving the problem of inconsistent color of the generated image in the prior art; introduces the new concept of "noise cluster" to judge the roughness of the generated image. The more noise clusters there are (or connected into pieces), the worse the picture of the generated image is, and the generation parameter combination should be rejected. The problem that there is no indicator to accurately measure the quality of the sand painting generated by the prior art is solved. At the same time, the entire algorithm of the present invention is encapsulated in a Python class, and the class is mounted on the HTTP request response function of Django to realize the server application. The problem that the prior art does not have a perfect user interaction system platform is solved.

[0149] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0150] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] Example 2

[0152] See also Fig.10 Embodiment 2 of the present invention further provides an image-to-sand painting processing device based on deep learning, comprising:

[0153] The picture loading module 001 is used to construct the style picture and the content picture into a new data set class by setting the data set class, and encapsulate the new data set class by setting the function to complete the picture loading process;

[0154] The image preprocessing module 002 is used to preprocess the loaded style image and the content image according to a set method to obtain the preprocessed style image and the content image;

[0155] Convolutional neural network building module 003, used to build a convolutional neural network;

[0156] A target image initialization generation module 004 is used to perform content processing on the pre-processed style image and the content image through the convolutional neural network to initialize and generate a target image;

[0157] A picture feature extraction module 005, used to extract style features and content features from the pre-processed style picture and content picture respectively;

[0158] The convolutional neural network training and processing module 006 is used to iteratively train the convolutional neural network according to the extracted style features and content features through loss function calculation to obtain the trained convolutional neural network; process the initialized target image through the trained convolutional neural network to obtain the processed target image;

[0159] An image post-processing module 007, configured to post-process the processed target image so that the processed target image retains the optical color characteristics of the content image;

[0160] The image output module 008 is used to output the target image after the post-processing to obtain the final target image.

[0161] In this embodiment, in the image loading module 001, in the process of constructing the style image and the content image into a new dataset class by setting the dataset class, and encapsulating the new dataset class by setting the function, the PyTorch dataset class Dataset is used as the parent class to construct a new dataset class DataSetOri, and the new dataset class is encapsulated by overriding the init constructor, getitem function and len function.

[0162] In this embodiment, in the image preprocessing module 002, in the process of preprocessing the loaded style image and the content image according to a set method, the set method includes trimming the size of the style image and the content image and standardizing and normalizing each pixel of the style image and the content image.

[0163] In this embodiment, in the target image initialization generation module 004, in the process of performing content processing on the pre-processed style image and the content image through the convolutional neural network to initialize the generation of the target image, the content of the content image is copied to the generated target image; through the iterative training processing of the convolutional neural network, the target image is transformed from the content image to a content and style mixed image.

[0164] In this embodiment, the convolutional neural network training and processing module 006 includes:

[0165] Extraction feature input submodule 061, used for inputting the extracted style feature and the content feature into the convolutional neural network;

[0166] A loss calculation submodule 062, used to calculate content loss and style loss through a loss function;

[0167] A total loss acquisition submodule 063 is used to obtain the total loss of each round of the convolutional neural network training according to the obtained content loss and style loss;

[0168] A back propagation calculation submodule 064 is used to calculate the influence of each parameter in the convolutional neural network on the total loss through back propagation;

[0169] The parameter weight optimization submodule 065 is used to optimize and update the parameter weights in the convolutional neural network according to the influence of each parameter in the convolutional neural network on the total loss.

[0170] In this embodiment, in the convolutional neural network training and processing module 006, in the process of iteratively training the convolutional neural network by calculating the loss function according to the extracted style features and content features, the content loss is calculated by using the square difference sum strategy; the style loss is calculated by using the Gram matrix strategy;

[0171] The calculation formula of the content loss is:

[0172]

[0173] Where, L conetnt is the content loss; F G and F I are content pictures and style pictures, and the superscript l is the specified color channel;

[0174] The calculation formula of the style loss is:

[0175]

[0176] In the formula, G [l] (k 1 , k 2 ) is the Gram matrix; G S [l] (k 1 , k 2 ) and G G [l] (k 1 , k 2 ) are the Gram matrices of the two pictures respectively; k 1 and k 2 The matrix G [l] Two channels in ; and k 1 and k 2 Neurons at the same location in the channel.

[0177] In this embodiment, in the image post-processing module 007, in the process of post-processing the processed target image, the RGB mean value and the standard deviation are calculated by the calRGBMean function and the calRGBStd function; the RGB mean value and the standard deviation are tensorized, multiplied and added with each pixel of the processed target image, and the post-processing of the processed target image is completed.

[0178] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0179] Example 3

[0180] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method of converting an image to a sand painting based on deep learning is stored. The program code includes instructions for executing the method of converting an image to a sand painting based on deep learning of embodiment 1 or any possible implementation thereof.

[0181] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0182] Example 4

[0183] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0184] The processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute an image-to-sand painting method based on deep learning in Example 1 or any possible implementation thereof.

[0185] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0187] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0188] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for converting images into sand paintings based on deep learning, characterized in that: include: The style image and the content image are constructed into a new data set class by setting the data set class, and the new data set class is encapsulated by setting the function to complete the image loading process; Preprocessing the loaded style picture and the content picture according to a set method to obtain the preprocessed style picture and the content picture; Constructing a convolutional neural network, performing content processing on the preprocessed style image and the content image through the convolutional neural network, and initializing and generating a target image; Extracting style features and content features from the preprocessed style image and content image respectively; According to the extracted style features and content features, the convolutional neural network is iteratively trained by loss function calculation to obtain the trained convolutional neural network; the initialized target image is processed by the trained convolutional neural network to obtain a processed target image; Post-processing the processed target image so that the processed target image retains the optical color characteristics of the content image; The target image after the post-processing is output to obtain a final target image.

2. The method of converting an image into a sand painting based on deep learning according to claim 1, characterized in that: In the process of constructing the style image and the content image into a new dataset class by setting the dataset class and encapsulating the new dataset class by setting the function, the PyTorch dataset class Dataset is used as the parent class to construct a new dataset class DataSetOri, and the new dataset class is encapsulated by overriding the init constructor, getitem function and len function.

3. The method of converting an image into a sand painting based on deep learning according to claim 2, characterized in that: In the process of preprocessing the loaded style image and the content image according to a set method, the set method includes trimming the size of the style image and the content image and standardizing and normalizing each pixel of the style image and the content image.

4. The method of converting an image into a sand painting based on deep learning according to claim 3, characterized in that: In the process of performing content processing on the preprocessed style image and the content image through the convolutional neural network to initialize the generation of the target image, copying the content of the content image to the generated target image; Through iterative training processing of the convolutional neural network, the target image is transformed from the content image into a content and style mixed image.

5. The method of converting an image into a sand painting based on deep learning according to claim 4, characterized in that: In the process of iteratively training the convolutional neural network according to the extracted style features and the content features through loss function calculation, the steps of iteratively training the convolutional neural network are: Inputting the extracted style features and content features into the convolutional neural network; Calculate content loss and style loss through loss function; Obtaining the total loss of each round of the convolutional neural network training according to the obtained content loss and the style loss; Calculate the impact of each parameter in the convolutional neural network on the total loss through back propagation; The parameter weights in the convolutional neural network are optimized and updated according to the influence of each parameter in the convolutional neural network on the total loss.

6. The method of converting an image into a sand painting based on deep learning according to claim 5, characterized in that: In the process of iteratively training the convolutional neural network according to the extracted style features and content features through loss function calculation, the content loss is calculated by using the square difference sum strategy; the style loss is calculated by using the Gram matrix strategy; The calculation formula of the content loss is: Where, L conetnt is the content loss; F G and F I are content pictures and style pictures, and the superscript l is the specified color channel; The calculation formula of the style loss is: In the formula, G [l] (k1, k2) is the Gram matrix; G s [l] (k1, k2) and G G [l] (k1, k2) are the Gram matrices of the two images respectively; k1 and k2 are matrices G [l] Two channels in ; and They are neurons at the same position of K1 and K2 channels respectively.

7. The method of converting an image into a sand painting based on deep learning according to claim 6, characterized in that: In the process of post-processing the processed target image, the RGB mean value and standard deviation are calculated by the calRGBMean function and the calRGBStd function; the RGB mean value and standard deviation are tensorized, multiplied and added with each pixel of the processed target image, and the post-processing of the processed target image is completed.

8. A deep learning-based image-to-sand painting processing device, using any one of the deep learning-based image-to-sand painting methods described in claims 1-7, characterized in that: include: The image loading module is used to construct the style image and the content image into a new data set class by setting the data set class, and encapsulate the new data set class by setting the function to complete the image loading process; An image preprocessing module, used to preprocess the loaded style image and the content image according to a set method to obtain the preprocessed style image and the content image; Convolutional neural network building module, used to build convolutional neural networks; A target image initialization generation module, used to perform content processing on the preprocessed style image and the content image through the convolutional neural network to initialize and generate a target image; A picture feature extraction module, used to extract style features and content features from the preprocessed style picture and content picture respectively; A convolutional neural network training and processing module, configured to iteratively train the convolutional neural network according to the extracted style features and content features through loss function calculation to obtain the trained convolutional neural network; and process the initialized target image through the trained convolutional neural network to obtain a processed target image; An image post-processing module, used for post-processing the processed target image so that the processed target image retains the optical color characteristics of the content image; The image output module is used to output the target image after the post-processing to obtain the final target image.

9. The image-to-sand painting processing device based on deep learning according to claim 8, characterized in that: The convolutional neural network training and processing module includes: An extraction feature input submodule, used for inputting the extracted style features and the content features into the convolutional neural network; The loss calculation submodule is used to calculate the content loss and style loss through the loss function; A total loss acquisition submodule, used to obtain the total loss of each round of the convolutional neural network training according to the obtained content loss and style loss; A back propagation calculation submodule, used to calculate the influence of each parameter in the convolutional neural network on the total loss through back propagation; The parameter weight optimization submodule is used to optimize and update the parameter weights in the convolutional neural network according to the influence of each parameter in the convolutional neural network on the total loss.

10. The image-to-sand painting processing device based on deep learning according to claim 9, characterized in that: In the convolutional neural network training and processing module, in the process of iteratively training the convolutional neural network according to the extracted style features and the content features, by calculating the loss function, the content loss is calculated by using the square difference sum strategy; the style loss is calculated by using the Gram matrix strategy; The calculation formula of the content loss is: Where, L conetnt is the content loss; F G and F I are content pictures and style pictures, and the superscript 1 indicates the specified color channel; The calculation formula of the style loss is: In the formula, G [l] (k1, k2) is the Gram matrix; G s [l] (k1, k2) and G G [l] (k1, k2) are the Gram matrices of the two images respectively; k1 and k2 are matrices G [l] Two channels in ; and They are neurons at the same position of K1 and K2 channels respectively.

Images