A sketch image translation method and system without network training

Abstract

The present invention discloses a sketch image translation method and system that does not require network training. The method comprises: inputting a user-drawn sketch and automatically normalizing it into a standard image; utilizing a weighted scoring mechanism to calculate the sketch structure loss, image fidelity loss, and total loss; selecting the initial basis vectors and weights of the network sample space; iterating latent variables according to defined training objectives and training strategies to obtain a user-satisfied sketch-fitting image, and temporarily storing the n most recently iterated images for the user to select; and finally, the user selects the image that best meets their needs and saves it. The present invention can use an existing trained GAN generation network to obtain a relatively good image, thereby reducing the network training cost and speeding up the user's feedback acquisition speed. Furthermore, through a multi-scoring mechanism combining a classifier and a discriminator, the generated image is more suitable for the sketch and can better meet the recognition of the generated image by most people.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to a sketch image translation method and system without network training. Background Art

[0002] Hand-drawn sketches Figure 1 have always been one of the important ways for humans to express their ideas. Their simplicity and abstraction can be accepted and used by most people. In recent years, due to the large-scale popularization of touch-screen devices, people have got rid of the limitation of traditional hand-drawing that requires pens and papers. Hand-drawn sketches have become an important human-computer interaction method. With the development of deep learning in images, a series of technologies related to sketches, such as sketch completion, sketch recognition, sketch translation, and sketch retrieval, have also developed rapidly.

[0003] Sketch image translation is a type of image generation task. People input hand-drawn sketches, and through the processing of neural networks, generate the corresponding images. However, due to the high abstraction of sketches, and everyone's concept of the same thing may not be the same, and the styles of the drawn sketches are also different, the field of sketch image translation still faces huge challenges.

[0004] Due to the increasing parameters of existing neural networks, the requirements for hardware devices are getting higher and higher. For individuals or small-scale groups, it may not be possible to bear the expensive cost of training a network from scratch. Therefore, how to use existing trained networks to achieve the desired generation effect is a current hot topic.

[0005] One of the current existing technologies, in the paper "FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization", a training-free text-to-image generation method FuseDream based on BigGAN and CLIP. This method does not train the generation network, but trains the latent variable input of the sample space of the GAN network, so as to achieve the function of generating corresponding images from the input text. The initialized M latent variable basis vectors generate images through the existing pre-trained BigGAN. The images and the text input by the user are matched and scored by CLIP, and the k (k << M) latent variable basis vectors with the highest scores are selected. Then, the selected basis vectors of the sample space and their weights are continuously iterated to obtain the image with the highest score. Users can obtain relatively reliable image return results without too long waiting time. The disadvantage of this method is that it can only handle the problem of text-to-image generation and cannot handle the problem of sketch image translation. Since only one scoring network is used, the network may be affected by its style and may be biased.

[0006] The second existing technique is the Sketch Your Own GAN, a network that fine-tunes StyleGAN2 to achieve customized sketch-to-image translation. Users input one or more sketches of animals or buildings in the same pose. Two discriminators simultaneously compete with the StyleGAN2 generator to train a network capable of generating images of the corresponding poses. One sketch structure discriminator distinguishes the differences between the structure of the generated image after conversion to the sketch and the structure of the user-input sketch. The other image discriminator ensures that the generated image is realistic and conforms to the distribution of the real sample space, without artifacts caused by the abstract nature of the sketch. However, this method has the disadvantage that it can only be trained for a specific animal or building shape at a time and is very time-consuming, meaning users cannot quickly obtain the desired image. Furthermore, it requires advanced hardware for long-term training. Furthermore, GAN network training is unstable; each time a customized item is requested, the GAN must be retrained, which is likely to cause corruption. Therefore, this method cannot be generalized for practical applications. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for sketch image translation that does not require network training. The main problems solved by the present invention are that existing GAN networks often require large network parameters to achieve satisfactory results, resulting in very high hardware costs; they cannot meet the user's immediacy needs; existing networks change very quickly, making it difficult for products to quickly retrain the network and quickly adapt to new networks; and many existing sketch image translation networks generate images that are significantly different from the sketches. Specifically, how can we reduce hardware costs while ensuring user satisfaction? How can we avoid the need to retrain the network and quickly adapt existing or new networks? And how can we improve the fit between network-generated images and sketches?

[0008] In order to solve the above problems, the present invention proposes a sketch image translation method without network training, which includes:

[0009] Input a sketch drawn by the user, initialize the sketch and the GAN generator respectively, and then automatically normalize the sketch into a standard form image S;

[0010] Using a weighted scoring mechanism that combines the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition, the sketch structure loss L1 and the image realism losses L2 and L3 are calculated respectively, and then weighted and summed to obtain the total loss L;

[0011] The latent variables randomly generated when the GAN generator sample space is initialized are input into the pre-trained GAN generator and jointly scored together with the standard form sketch image S through the weighted scoring mechanism. The m latent variables with the highest scores are defined as the basis vectors ζ of the sample space. i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space;

[0012] The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the desired basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose;

[0013] After the iteration is completed, the final sketch translation result is obtained. The user can select and save the image that best meets his or her needs from the image with the best network score after the iteration and the n images temporarily stored in the most recent iteration.

[0014] Preferably, a sketch drawn by a user is input, the sketch and the GAN generator are initialized respectively, and then the sketch is automatically normalized into a standard form image S, specifically:

[0015] The user draws a sketch on the device and chooses to submit it to the GAN;

[0016] Initialize the hand-drawn sketch and normalize the hand-drawn sketch into a standard form 256*256*3 image S;

[0017] Initialize the GAN generator, which uses BigGAN. The output of the last layer of the GAN generator uses the hyperbolic tangent function Tanh as the activation function, and finally obtains a 256*256*3 color image;

[0018] The BigGAN is a GAN generator that outputs a 256*256*3 format. It consists of one Linear layer and six ResBlock layers. The Linear layer converts the input noise z into 4*4*16 features, which are then fed into the next ResBlock layer. Each ResBlock layer upsamples the input features by a factor of 2 and performs a series of convolutional layers, ReLU activation functions, and batch normalization. The residuals are then added to the upsampled input of the previous layer to generate features that can be fed into the next layer.

[0019] Preferably, a weighted scoring mechanism combining the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition is used to calculate the sketch structure loss L1 and the image realism losses L2 and L3 respectively, and then weighted and summed to obtain the total loss L, specifically:

[0020] Sketch structure discrimination loss definition, input the latent variable to the image S generated by the pre-trained GAN generator g , S g As a condition input Photo-sketching network, get its sketch outline O g ; The obtained sketch outline O g and the normalized user input sketch S, respectively, are passed through the encoder part of the pre-trained sketch recognition network Multigraphtransformer to obtain the features f of the two. g and f s , the feature length is the number of classifications q of the recognition network, and the mean square error of the features is calculated to obtain the loss. The specific calculation formula is as follows:

[0021] L1=|f s -f g | 2 / q,

[0022] The purpose of calculating this loss is to make the contour of the generated image fit the structure of the user's hand-drawn sketch, but not to rely entirely on the overfitting problem caused by pixel-level differences;

[0023] The image realistic discrimination loss is defined as follows: the generated image S g Input them into the pre-trained discriminators D2 and D3 of BigGAN and StyleGAN2 respectively to obtain the losses L2 and L3. The specific calculation formulas are as follows:

[0024] L k =-log D k (G(z)),k=2,3

[0025] Among them, G(z) is an image tensor generated by using the latent variable z as the input of the GAN generator. The purpose of calculating this loss is to make the generated image more conform to the distribution of real images, so as to make the generated image more realistic. The purpose of using two discriminators is to prevent the discriminator of a single network from restricting the style of the generated image to the style of real images learned;

[0026] Definition of weighted total loss. Combining the sketch structure loss L1 and the image realism losses L2 and L3, the total loss is defined as:

[0027] L = L1 + αL2 + βL3,

[0028] Among them, the hyperparameters are set as α = β = 0.5;

[0029] Among them, the sketch recognition network Multigraph transformer is a network of the transformer model composed of an encoder and a decoder. The number of input sequences in its encoder part is equal to the number of its feature outputs. Since the self-attention mechanism can ignore the positional relationship of the input sequences, only the penultimate layer of the output layer of the first sequence needs to be taken, that is, the corresponding feature of the layer before the classifier Softmax layer. This feature encompasses all the information of the entire sketch. The sketch recognition network can regard the input sketch as a line formed by a series of points, and using the self-attention mechanism to learn the sequence of strokes can effectively identify the structural features of the sketch.

[0030] Preferably, the latent variables randomly generated during the initialization of the GAN generator sample space are input into the pre-trained GAN generator, and jointly scored with the sketch image S in the standard form through the weighted scoring mechanism. The m latent variables with the highest scores are defined as the basis vectors ζ of the sample space i And determine the initialization weights to complete the selection of the initial basis vectors and weights of the network sample space, specifically:

[0031] During the initialization of the GAN generator sample space, the GAN generator randomly generates M (default setting M = 10000) different latent variables z and inputs them into the pre-trained GAN generator. The latent variables pass through the GAN generator to generate M images with a resolution of 256*256;

[0032] 3]After jointly scoring the M images generated by the M latent variables with the sketch S hand-drawn by the user through the weighted scoring mechanism, m (m << M, default setting m = 15) latent variables with the highest scores, that is, the lowest loss, are selected. These latent variables are defined as the basis vectors ζ of the sample space i , and the initialization weight is the weight of each basis vector That is, the initialized hidden variable, that is, the network input vector. The specific calculation formula is:

[0033]

[0034] Preferably, the training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the desired basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), and then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose, specifically:

[0035] Define the training goal: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in step 1 is minimized;

[0036] Define the training strategy: The optimizer uses the Adam optimizer, and the Adam hyperparameters are set to β1 = 0.9, β2 = 0.999. The learning rate uses a staged descent learning rate. The initial learning rate is set to 0.001, and it is reduced to 0.1 times the original rate every 500 iterations. Using the training strategy, the desired basis vector ζ is found by gradual gradient descent. i And its corresponding weight ω i (i=1, 2, ..., m);

[0037] Iterative hidden variables: Starting from the initial basis vectors and weights of the selected network sample space, the image S generated by the GAN generator g , after the combined loss defined by the weighted total loss, the basis vectors and weights in the input vector are iterated according to the defined training objectives and training strategies, and an image that better fits the sketch is gradually obtained. The number of iterations is k times, which can be determined by the user. The user can manually stop when a satisfactory image is obtained, otherwise k = 1000 is the default; the n images of the most recent iteration (n = 15) are temporarily stored, and images more than n iterations ago will be deleted to reduce the memory burden. The temporarily stored images are available for user selection after the iteration ends.

[0038] Accordingly, the present invention also provides a sketch image translation system that does not require network training, comprising:

[0039] A sketch input and normalization unit, which inputs a sketch drawn by a user, initializes the sketch and the GAN generator respectively, and then automatically normalizes the sketch into a standard form image S;

[0040] A sketch structure loss calculation unit is configured to calculate the sketch structure loss L1 and the image realism loss L2 and L3 respectively using a weighted scoring mechanism that combines the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition, and then weight and sum them to obtain a total loss L;

[0041] The network sample space initialization vector and weight selection unit is used to input the latent variables randomly generated when the GAN generator sample space is initialized into the pre-trained GAN generator, and jointly score them together with the standard form sketch image S through the weighted scoring mechanism, and obtain the m highest-scoring latent variables defined as the basis vectors ζ of the sample space. i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space;

[0042] Image training and fitting sketch output unit, used to define the training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the desired basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose;

[0043] The final image output unit is used to obtain the final sketch translation result after the iteration is completed. The user can select the image that best meets his or her needs from the image with the best network score after the iteration and the n images temporarily stored in the most recent iteration to save.

[0044] The implementation of the present invention has the following beneficial effects:

[0045] The present invention can realize sketch-image translation without training the network, and can use the existing trained GAN generation network to obtain relatively good images, thereby greatly reducing the training cost of the network. The existing sketch-image translation with good results requires re-fine-tuning and training of the pre-trained GAN network, which greatly affects the time it takes for users to obtain the desired image. The method provided by the present invention greatly speeds up the speed at which users obtain feedback. The multi-scoring mechanism combining the classifier and the discriminator proposed in the present invention makes the generated image more suitable for the sketch, and the generated image also reduces the probability of being confused and unrecognizable due to the abstractness of the sketch, which can better meet the recognition of the generated image by most people. The present invention saves the collection of data sets, and by using the pre-trained GAN network for image generation, the cost of collecting sketch data sets can be saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of a sketch image translation method without network training according to an embodiment of the present invention;

[0047] Figure 2 4 is a structural diagram of a sketch image translation system without network training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] Figure 1 FIG is a flowchart of a sketch image translation method without network training according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0050] S1, inputs a sketch drawn by the user, initializes the sketch and the GAN generator respectively, and then automatically normalizes the sketch into a standard form image S;

[0051] S2, using a weighted scoring mechanism that combines the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition, respectively calculates the sketch structure loss L1 and the image realism losses L2 and L3, and then weights and sums them to obtain the total loss L;

[0052] S3: Input the latent variables randomly generated when the GAN generator sample space is initialized into the pre-trained GAN generator, and jointly score them together with the standard form sketch image S through the weighted scoring mechanism. The m latent variables with the highest scores are defined as the basis vectors ζ of the sample space. i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space;

[0053] S4, define the training goal as finding a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the desired basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose;

[0054] S5, after the iteration is completed, the final sketch translation result is obtained, and the user can select the image that best meets his or her needs from the image with the best network score after the iteration and the n images temporarily stored in the most recent iteration to save.

[0055] Step S1 is as follows:

[0056] S1-1, the user draws a sketch on the device and chooses to submit it to the GAN;

[0057] S1-2, initializing the hand-drawn sketch and normalizing the hand-drawn sketch into a standard form 256*256*3 image S;

[0058] S1-3, initialize the GAN generator, the GAN generator uses BigGAN, and the output of the last layer of the GAN generator uses the hyperbolic tangent function Tanh as the activation function, and finally obtains a 256*256*3 color image;

[0059] S1-3-1, the BigGAN is a GAN generator that outputs a 256*256*3 pattern. It consists of one Linear layer and six ResBlock layers. The Linear layer converts the input noise z into 4*4*16 features, which are then input to the next ResBlock layer. Each ResBlock layer upsamples the input features by a factor of 2 and performs a series of convolutional layers, ReLU activation functions, and batch normalization. The residuals are then added to the upsampled input of the previous layer to generate features that can be input to the next layer.

[0060] Step S2 is as follows:

[0061] S2-1, the definition of the sketch structure discrimination loss, inputs the latent variable into the image S generated by the pre-trained GAN generator g , S g As a condition input Photo-sketching network, get its sketch outline O g ; The obtained sketch outline O g and the normalized user input sketch S, respectively, are passed through the encoder part of the pre-trained sketch recognition network Multigraphtransformer to obtain the features f of the two. g and f s , the feature length is the number of classifications q of the recognition network, and the mean square error of the features is calculated to obtain the loss. The specific calculation formula is as follows:

[0062] L1=|f s -f g | 2 / q,

[0063] The purpose of calculating this loss is to make the contour of the generated image fit the structure of the user's hand-drawn sketch, but not to rely entirely on the overfitting problem caused by pixel-level differences;

[0064] S2-1-1, the photo-sketching network is a cGAN type network that takes an image as a conditional input, and its generator can generate a sketch outline corresponding to the conditional image;

[0065] S2-1-2. The sketch recognition network Multigraph transformer is a network of the transformer model consisting of an encoder and a decoder. The number of input sequences in its encoder part is equal to the number of its feature outputs. Since the self-attention mechanism can ignore the positional relationship of the input sequences, only the penultimate layer of the output layer of the first sequence, that is, the layer before the classifier Softmax layer, is taken. The corresponding features include all the information of the entire sketch. The sketch recognition network can regard the input sketch as a line formed by a series of points. Learning the sequence of strokes with the self-attention mechanism can effectively identify the structural features of the sketch;

[0066] S2-2. Definition of the image realism discrimination loss. Input the generated image S g into the pre-trained discriminators D2 and D3 of BigGAN and StyleGAN2 respectively to obtain losses L2 and L3. The specific calculation formula is as follows:

[0067] L k =-log D k (G(z)), k = 2, 3,

[0068] The purpose of calculating this loss is to make the generated image more in line with the distribution of real images, so that the generated image is more realistic. The purpose of using two discriminators is to prevent the discriminator of a single network from restricting the style of the generated image to the style of real images learned;

[0069] S2-3. Definition of the weighted total loss. Combining the sketch structure loss L1 and the image realism losses L2 and L3, the total loss is defined as:

[0070] L = L1 + αL2 + βL3,

[0071] where the hyperparameters are set as α = β = 0.5.

[0072] Step S3 is as follows:

[0073] S3-1. When initializing the sample space of the GAN generator, the GAN generator randomly generates M (default setting M = ********) different latent variables z and inputs them into the pre-trained GAN generator. The latent variables pass through the GAN generator to generate M images with a resolution of 256*256;

[0074] S3-2. After jointly scoring the user-drawn sketch S and the M images generated by the M latent variables through the weighted scoring mechanism, m (m << M, default setting m = 15) latent variables with the highest scores, that is, the lowest losses, are obtained. These latent variables are selected and defined as the basis vectors ζ of the sample space i, initialize the weights to the weights of each basis vector That is, the initialized hidden variable, that is, the network input vector. The specific calculation formula is:

[0075]

[0076] Step S4 is as follows:

[0077] S4-1, define the training objectives: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in step 1 is minimized;

[0078] S4-2, define the training strategy: the optimizer uses the Adam optimizer, the Adam hyperparameters are set to β1 = 0.9, β2 = 0.999, the learning rate uses a staged descent learning rate, the initial learning rate is set to 0.001, and it is reduced to 0.1 times the original rate every 500 iterations. Using the training strategy, the desired basis vector ζ is found by gradual gradient descent. i And its corresponding weight ω i (i=1, 2, ..., m);

[0079] S4-3, iterative latent variables: starting from the initial basis vectors and weights of the selected network sample space, the image S generated by the GAN generator g , after the combined loss defined by the weighted total loss, the basis vectors and weights in the input vector are iterated according to the defined training objectives and training strategies, and an image that better fits the sketch is gradually obtained. The number of iterations is k times, which can be determined by the user. The user can manually stop when a satisfactory image is obtained, otherwise k = 1000 is the default; the n images of the most recent iteration (n = 15) are temporarily stored, and images more than n iterations ago will be deleted to reduce the memory burden. The temporarily stored images are available for user selection after the iteration ends.

[0080] Accordingly, the present invention also provides a sketch image translation system that does not require network training, such as Figure 2 Shown, including:

[0081] A sketch input and normalization unit 1 is used to input a sketch drawn by a user, initialize the sketch and the GAN generator respectively, and then automatically normalize the sketch into a standard image S using the GAN generator;

[0082] Specifically, the user draws a sketch on the device and chooses to submit it to the GAN;

[0083] Initialize the hand-drawn sketch and normalize it into a standard 256*256*3 image S. Initialize the GAN generator, using BigGAN. The output of the last layer of the GAN generator uses the hyperbolic tangent function Tanh as the activation function, ultimately obtaining a 256*256*3 color image. The BigGAN is a GAN generator that outputs a 256*256*3 pattern and consists of one Linear layer and six ResBlock layers. The Linear layer converts the input noise z into 4*4*16 features, which are then input to the next ResBlock layer. Each ResBlock layer upsamples the input features by a factor of 2 and performs a series of convolutional layers, ReLU activation functions, and BN normalization. The residuals are then added to the upsampled input of the previous layer to obtain features that can be input to the next layer.

[0084] Sketch structure loss calculation unit 2, configured to calculate the sketch structure loss L1 and the image realism loss L2 and L3 respectively using a weighted scoring mechanism combining the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition, and then weight and sum them to obtain the total loss L;

[0085] Specifically, the sketch structure discrimination loss is defined as follows: the latent variable is input into the image S generated by the pre-trained GAN generator g , S g As a condition input Photo-sketching network, get its sketch outline O g ; The obtained sketch outline O g and the normalized user input sketch S, respectively, are passed through the encoder part of the pre-trained sketch recognition network Multigraphtransformer to obtain the features f of the two. g and f s , the feature length is the number of classifications q of the recognition network, and the mean square error of the feature is calculated to obtain the loss L1=|f s -f g | 2 / q, the purpose of calculating this loss is to make the outline of the generated image fit the structure of the user's hand-drawn sketch, but not completely rely on the overfitting problem caused by pixel-level differences; the definition of image realistic discrimination loss is to convert the generated image S g Input into the pre-trained discriminators D2 and D3 of BigGAN and StyleGAN2 respectively, and obtain the losses L2 and L3, i.e. L k =-log D k(G(z)), where k = 2, 3. Here, G(z) is an image tensor generated by using the latent variable z as the input of the GAN generator. The purpose of calculating this loss is to make the generated image more conform to the distribution of real images, so that the generated image is more realistic. The purpose of using two discriminators is to prevent the discriminator of a single network from restricting the style of the generated image to the style of real images learned. The weighted total loss is defined as follows: combining the sketch structure loss L1 and the image realism losses L2 and L3, the total loss is defined as L = L1 + αL2 + βL3, where the hyperparameters are set as α = β = 0.5. The Photo - sketching network is a network of the cGAN type, which takes an image as the conditional input, and its generator can generate the sketch contour corresponding to the conditional image. The sketch recognition network Multigraph transformer is a network of the transformer model composed of an encoder and a decoder. The number of input sequences in its encoder part is equal to the number of its feature outputs. Since the self - attention mechanism can ignore the positional relationship of the input sequences, only the penultimate layer of the output layer of the first sequence needs to be taken, that is, the corresponding feature of the layer before the classifier Softmax layer. This feature encompasses all the information of the entire sketch. The sketch recognition network can regard the input sketch as a line formed by a series of points, and using the self - attention mechanism to learn the sequence of strokes can effectively identify the structural features of the sketch.

[0086] The network sample space initial vector and weight selection unit 3 is used to input the latent variable randomly generated during the initialization of the GAN generator sample space into the pre - trained GAN generator, and jointly score it with the standard - form sketch image S through the weighted scoring mechanism, and define the m latent variables with the highest scores as the basis vectors ζ of the sample space i And determine the initial weights to complete the selection of the initial basis vectors and weights of the network sample space;

[0087] Specifically, when initializing the GAN generator sample space, the GAN generator randomly generates M (default setting M = 10000) different latent variables z and inputs them into the pre - trained GAN generator. The latent variables pass through the GAN generator to generate M images with a resolution of 256 * 256. After jointly scoring the M images generated by the M latent variables with the user - hand - drawn sketch S through the weighted scoring mechanism, m (m << M, default setting m = 15) latent variables with the highest scores, that is, the lowest losses, are obtained. These latent variables are selected and defined as the basis vectors ζ of the sample space i , and the initial weights are the weights of each basis vector That is, the initialized latent variable, which is also the network input vector. The specific calculation formula is:

[0088] Image training and fitting sketch output unit 4, used to define the training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in step 1 is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to gradually descend the gradient to find the basis vector ζ that is desired to be found i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variables, and temporarily store the n images of the most recent iteration for the user to choose;

[0089] Specifically, define the training objectives: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in step 1 is minimized; define the training strategy: the optimizer uses the Adam optimizer, the Adam hyperparameters are set to β1=0.9, β2=0.999, the learning rate uses a staged descent learning rate, the initial learning rate is set to 0.001, and it is reduced to 0.1 times the original rate every 500 iterations. Using the training strategy, the desired basis vector ζ is found by gradual gradient descent. i And its corresponding weight ω i (i=1, 2, ..., m); Iterative hidden variables: Starting from the initial basis vectors and weights of the selected network sample space, the image S generated by the GAN generator g , after the combined loss defined by the weighted total loss, the basis vectors and weights in the input vector are iterated according to the defined training objectives and training strategies, and an image that better fits the sketch is gradually obtained. The number of iterations is k times, which can be determined by the user. The user can manually stop when a satisfactory image is obtained, otherwise k = 1000 is the default; the n images of the most recent iteration (n = 15) are temporarily stored, and images more than n iterations ago will be deleted to reduce the memory burden. The temporarily stored images are available for user selection after the iteration ends.

[0090] The final image output unit 5 is used to obtain the final sketch translation result after the iteration is completed. The user can select the image that best meets his or her needs from the image with the best network score after the iteration and the n images temporarily stored in the most recent iteration to save.

[0091] Therefore, the present invention can achieve sketch image translation without training the network, and can use the existing trained GAN generation network to obtain better images, thereby greatly reducing the training cost of the network. The method provided by the present invention greatly speeds up the speed at which users obtain feedback. The multi-scoring mechanism combining the classifier and the discriminator proposed in the present invention makes the generated image more suitable for the sketch, and the generated image also reduces the probability of being confused and unrecognizable due to the abstractness of the sketch, which can better meet the recognition of the generated image by most people. The present invention saves the collection of data sets. By using a pre-trained GAN network for image generation, the cost of collecting sketch data sets can be saved.

[0092] The above is a detailed introduction to the sketch image translation method and system without network training provided in the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A sketch image translation method without network training, characterized in that: The method comprises: Input a sketch drawn by the user, initialize the sketch and the GAN generator respectively, and then automatically normalize the sketch into a standard form image S; Using a weighted scoring mechanism that combines the sketch structure discrimination definition, image realism discrimination loss definition, and weighted total loss definition, the sketch structure loss L1 and image realism losses L2 and L3 are calculated separately, and then weighted and summed to obtain the total loss L; The latent variables randomly generated when the GAN generator sample space is initialized are input into the pre-trained GAN generator and jointly scored together with the standard form image S through the weighted scoring mechanism. The m latent variables with the highest scores are defined as the basis vectors ζ of the sample space. i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space; The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose; After the iteration is completed, the final sketch translation result is obtained. The user selects the image that best meets their needs from the image with the best network score after the iteration and the n images of the most recent iteration stored temporarily and saves it; Specifically, the weighted scoring mechanism that combines the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition is used to calculate the sketch structure loss L1 and the image realism losses L2 and L3 respectively, and then weight and sum them to obtain the total loss L, which is specifically: Sketch structure discrimination loss definition, input the latent variable to the image S generated by the pre-trained GAN generator g , S g As a condition input Photo-sketching network, get its sketch outline O g ; The obtained sketch outline O g and the standard form image S, respectively, obtain the features f of the two through the encoder part of the pre-trained sketch recognition network Multigraph transformer g and f s , the feature length is the number of classifications q of the recognition network, and the mean square error of the features is calculated to obtain the loss. The specific calculation formula is as follows: L1=|f s -f g | 2 / q; The image realistic discrimination loss is defined as follows: the generated image S g Input them into the pre-trained discriminators D2 and D3 of BigGAN and StyleGAN2 respectively to obtain the losses L2 and L3. The specific calculation formulas are as follows: Lk=-logDk(G(z)),k=2,3 Where G(z) is the image tensor generated using the hidden variable z as the input of the GAN generator; The weighted total loss definition combines the sketch structure loss L1 and the image realism losses L2 and L3 to define the total loss as: L=L1+αL2+βL3, Among them, the hyperparameter is set to α=β=0.5; Among them, the sketch recognition network Multigraph transformer is a network of transformer models consisting of an encoder and a decoder. The number of input sequences of its encoder part is equal to the number of its feature outputs. Since the self-attention mechanism can ignore the positional relationship of the input sequences, it only needs to take the corresponding features of the second-to-last layer of the output layer of the first sequence, that is, the layer before the classifier Softmax layer. This feature encompasses all the information of the entire sketch. The sketch recognition network regards the input sketch as a line connected by a series of points. Using the self-attention mechanism to learn the stroke sequence can effectively identify the structural features of the sketch.

2. The sketch image translation method without network training as claimed in claim 1, characterized in that: The user-drawn sketch is input, the sketch and the GAN generator are initialized respectively, and then the sketch is automatically normalized into a standard image S, specifically: The user draws a sketch on the device and chooses to submit it to the GAN; Initialize the hand-drawn sketch and normalize the hand-drawn sketch into a standard form 256*256*3 image S; Initialize the GAN generator, which uses BigGAN. The output of the last layer of the GAN generator uses the hyperbolic tangent function Tanh as the activation function, and finally obtains a 256*256*3 color image; Among them, the BigGAN is a GAN generator that outputs a 256*256*3 style. It consists of a Linear layer and six ResBlock layers. The Linear layer converts the input noise z into a 4*4*16 feature, which is then input to the next ResBlock layer; each ResBlock layer performs an operation including a 2x upsampling process and a series of convolution layers, ReLU activation function and BN normalization on the input features, and then adds the residual of the upsampled result with the input of the previous layer to obtain features that can be input to the next layer.

3. The sketch image translation method without network training as claimed in claim 1, characterized in that: The latent variables randomly generated when the GAN generator sample space is initialized are input into the pre-trained GAN generator and jointly scored together with the standard form image S through the weighted scoring mechanism. The m latent variables with the highest scores are defined as the basis vectors ζ of the sample space. i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space, specifically: When initializing the GAN generator sample space, the GAN generator randomly generates M different latent variables z and inputs them into the pre-trained GAN generator. The latent variables pass through the GAN generator to generate M 256*256 resolution images; After jointly scoring the image S in the standard form and the M images generated by the M different latent variables through the weighted scoring mechanism, m (m << M) latent variables with the highest scores, that is, the lowest losses, are obtained. These latent variables are selected and defined as the basis vectors ζ of the sample space. i , initialize the weight as the weight of each basis vector That is, the initialized latent variable, which is also the network input vector. The specific calculation formula is:

4. The sketch image translation method without network training as claimed in claim 1, characterized in that: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), and then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose, specifically: Define the training goal: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in step 1 is minimized; Define the training strategy: The optimizer uses the Adam optimizer, and the Adam hyperparameters are set to β1 = 0.9, β2 = 0.

999. The learning rate uses a staged descent learning rate. The initial learning rate is set to 0.001, and it is reduced to 0.1 times the original value every 500 iterations. Using the training strategy, the basis vector ζ is found by gradual gradient descent. i And its corresponding weight ω i (i=1, 2, ..., m); Iterative hidden variables: Starting from the initial basis vectors and weights of the selected network sample space, the image S generated by the GAN generator g , after the combined loss defined by the weighted total loss, the basis vectors and weights in the input vector are iterated according to the defined training objectives and training strategies, and an image that better fits the sketch is gradually obtained. The number of iterations is k times, which can be determined by the user. The user can manually stop when a satisfactory image is obtained, otherwise it is executed according to the default number of k times; the n images of the most recent iteration are temporarily stored, and images more than n iterations ago will be deleted to reduce the memory burden. The temporarily stored images are available for user selection after the iteration ends.

5. A sketch image translation system that does not require network training, characterized by: The system comprises: A sketch input and normalization unit, which inputs a sketch drawn by a user, initializes the sketch and the GAN generator respectively, and then automatically normalizes the sketch into a standard form image S; The sketch structure loss calculation unit is used to calculate the sketch structure loss L1 and the image realism loss L2 and L3 respectively using a weighted scoring mechanism that combines the sketch structure discrimination definition, the image realism discrimination loss definition, and the weighted total loss definition, and then weight and sum them to obtain the total loss L; The network sample space initialization vector and weight selection unit is used to input the latent variables randomly generated when the GAN generator sample space is initialized into the pre-trained GAN generator, and jointly score them together with the standard form image S through the weighted scoring mechanism, and obtain the m highest-scoring latent variables defined as the basis vectors ζ of the sample space i And determine the initialization weights to complete the initial basis vector and weight selection of the network sample space; Image training and fitting sketch output unit, used to define the training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the total loss L is minimized; the training strategy is defined as using the Adam optimizer and a staged descent learning rate to find the basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m), then according to the defined training objectives and training strategies, obtain the image of the fitting sketch that the user is satisfied with by iterating the latent variable z, and temporarily store the n images of the most recent iteration for the user to choose; A final image output unit is used to obtain the final sketch translation result after the iteration is completed. The user selects the image that best meets their needs from the image with the best network score after the iteration and the n images of the most recent iteration stored temporarily for storage; Specifically, the sketch structure loss calculation unit requires the definition of sketch structure discrimination loss, inputs the latent variable into the image S generated by the pre-trained GAN generator g , S g As a condition input Photo-sketching network, get its sketch outline O g ; The obtained sketch outline O g and the standard form image S, respectively, obtain the features f of the two through the encoder part of the pre-trained sketch recognition network Multigraph transformer g and f s , the feature length is the number of classifications q of the recognition network, and the mean square error of the feature is calculated to obtain the loss L1=|f s -f g | 2 / q; Definition of image realistic discrimination loss, the generated image S g Input into the pre-trained discriminators D2 and D3 of BigGAN and StyleGAN2 respectively, and obtain the losses L2 and L3, that is, Lk = -logDk(G(z)), k = 2, 3, where G(z) is the image tensor generated by using the hidden variable z as the input of the GAN generator; the weighted total loss definition, combined with the sketch structure loss L1 and the image realistic losses L2 and L3, defines the total loss as L = L1 + αL2 + βL3, where the hyperparameter is set to α = β = 0.5; where the sketch recognition network Multigraph The transformer is a network of transformer models consisting of an encoder and a decoder. The number of input sequences in the encoder is equal to the number of its feature outputs. Since the self-attention mechanism can ignore the positional relationship of the input sequences, it only needs to take the corresponding features of the second-to-last layer of the output layer of the first sequence, that is, the layer before the classifier Softmax layer. This feature encompasses all the information of the entire sketch. The sketch recognition network regards the input sketch as a series of lines connected by points. Using the self-attention mechanism to learn the stroke sequence can effectively identify the structural features of the sketch.

6. The sketch image translation system without network training as claimed in claim 5, characterized in that: The sketch input and normalization unit requires the user to hand-draw a sketch on the device and choose to submit it to the GAN; initialize the hand-drawn sketch and normalize the hand-drawn sketch into a standard 256*256*3 image S; initialize the GAN generator, the GAN generator uses BigGAN, and the output of the last layer of the GAN generator uses the hyperbolic tangent function Tanh as the activation function, and finally obtains a 256*256*3 color image; wherein, the BigGAN is a GAN generator that outputs a 256*256*3 style, which consists of a Linear layer and six ResBlock layers. The Linear layer converts the input noise z into a 4*4*16 feature, which is then input to the next ResBlock layer; each ResBlock layer performs operations including a 2x upsampling process and a series of convolution layers, ReLU activation function and BN normalization on the input feature, and then performs residual addition on the result after upsampling with the input of the previous layer to obtain features that can be input to the next layer.

7. The sketch image translation system without network training as claimed in claim 5, characterized in that: The network sample space initial vector and weight selection unit, when initializing the sample space of the GAN generator, requires the GAN generator to randomly generate M different latent variables z and input them into the pre-trained GAN generator. The latent variables pass through the GAN generator to generate M images with a resolution of 256*256. After jointly scoring the standard-form image S and the M images generated by the M different latent variables through the weighted scoring mechanism, m (m << M) latent variables with the highest scores, that is, the lowest losses, are obtained. These latent variables are selected and defined as the basis vectors ζ of the sample space i , initialize the weight as the weight of each basis vector That is, the initialized latent variable, which is also the network input vector, is 8. The sketch image translation system without network training as claimed in claim 5, characterized in that: The image training and fitting sketch output unit requires defining the training target: The training goal is to find a set of basis vectors ζ i And its corresponding weight ω i (i=1, 2, ..., m), so that the weighted loss L defined in the step is minimized; define the training strategy: the optimizer uses the Adam optimizer, the Adam hyperparameters are set to β1=0.9, β2=0.999, the learning rate uses a staged descent learning rate, the initial learning rate is set to 0.001, and it is reduced to 0.1 times the original every 500 iterations. Use the training strategy to find the basis vector ζ by gradual gradient descent i And its corresponding weight ω i (i=1, 2, ..., m); Iterative hidden variables: Starting from the initial basis vectors and weights of the selected network sample space, the image S generated by the GAN generator g , after the combined loss defined by the weighted total loss, the basis vectors and weights in the input vector are iterated according to the defined training objectives and training strategies, and an image that better fits the sketch is gradually obtained. The number of iterations is k times, which can be determined by the user. The user can manually stop when a satisfactory image is obtained, otherwise it is executed according to the default number of k times; the n images of the most recent iteration are temporarily stored, and images more than n iterations ago will be deleted to reduce the memory burden. The temporarily stored images are available for user selection after the iteration ends.

Images