Image Conversion Method, Device, Equipment and Medium Based on Small Data Generation Network

Abstract

The present invention discloses an image conversion method, device, equipment and medium based on a small data generation network. The method includes: iteratively training an initial image conversion model according to a large data image set to obtain a trained image conversion model, training an encoder according to the large data image set to obtain a target encoder and configuring parameters for the image conversion model, optimizing and training the configured image conversion model according to a small data cartoon image set to obtain an image conversion model with a cartoon style, and performing style conversion on the input image to be converted according to the image conversion model to obtain a target cartoon image. The present invention belongs to the technical field of image style conversion. By modifying the training method of the existing generation network and optimizing and training the image conversion model through the small data cartoon image set, it is possible to train the generation network with small training data to perform style conversion on images, and solve the problem that the encoder cannot be applied to small training data during the training process of the unconditional generation network.

Description

Technical Field

[0001] The present invention relates to the technical field of image style conversion, and in particular to an image conversion method, device, equipment and medium based on a small data generation network. Background Art

[0002] Portraits are usually required in chat software or dating software to reflect the personal characteristics of users. Existing technical methods can intelligently generate cartoon portraits based on user images for users to use in chat software or dating software. However, when training the model for generating cartoon portraits, it is usually difficult to collect a large amount of training data. Since the cartoon portrait is the target information that the model needs to generate and output, the amount of data of the made or collected cartoon faces is very small. The lack of training data makes it impossible to effectively train the deep learning model, and the effect of the cartoon images generated by the model based on the input real face images is very poor. Although the conditional generative adversarial network (conditional GAN) can solve the problem of less training data, the generation effect of the conditional generative adversarial network is much worse than that of the unconditional generative adversarial network (unconditional GAN). Although the existing unconditional generative adversarial network can enable the model to generate picture styles (such as cartoons) with a small amount of data from random variables, and the effect of generating cartoon images is good, the unconditional generative adversarial network cannot implement the function of image conversion under a small data sample size. Therefore, the existing technical methods have the problem that the generation network cannot be trained based on small data samples to achieve image conversion. Summary of the Invention

[0003] Embodiments of the present invention provide an image conversion method, device, equipment and medium based on a small data generation network, aiming to solve the problem in the existing technical methods that the generation network cannot be trained based on small data samples to achieve image conversion.

[0004] In a first aspect, embodiments of the present invention provide an image conversion method based on a small data generation network, and the method includes:

[0005] Iteratively training an initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model;

[0006] Training a preset encoder according to a multi-layer analysis network in the image conversion model and the large data image set to obtain a corresponding target encoder;

[0007] Configuring parameters of the image conversion model according to the target encoder to obtain the configured image conversion model;

[0008] Optimize and train the configured image conversion model according to the preset training rules and the pre-stored small data cartoon image set to obtain an image conversion model with a cartoon style;

[0009] If the input image to be converted is received, perform style conversion on the image to be converted according to the image conversion model with a cartoon style to obtain a target cartoon image corresponding to the image to be converted.

[0010] In a second aspect, an embodiment of the present invention provides an image conversion device based on a small data generation network, which includes:

[0011] An initial image conversion model training unit, configured to iteratively train an initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model;

[0012] An encoder training unit, configured to train a preset encoder according to the multi-layer analysis network in the image conversion model and the large data image set to obtain a corresponding target encoder;

[0013] A parameter configuration unit, configured to configure parameters of the image conversion model according to the target encoder to obtain the configured image conversion model;

[0014] A model optimization training unit, configured to optimize and train the configured image conversion model according to the preset training rules and the pre-stored small data cartoon image set to obtain an image conversion model with a cartoon style;

[0015] A target cartoon image acquisition unit, configured to, if the input image to be converted is received, perform style conversion on the image to be converted according to the image conversion model with a cartoon style to obtain a target cartoon image corresponding to the image to be converted.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the image conversion method based on a small data generation network described in the first aspect above is implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the image conversion method based on a small data generation network described in the first aspect above.

[0018] An embodiment of the present invention provides an image conversion method, device, equipment and medium based on a small-data generation network. The initial image conversion model is iteratively trained according to a large-data image set to obtain a trained image conversion model. The encoder is trained according to the large-data image set to obtain a target encoder and the parameters of the image conversion model are configured. Then, the configured image conversion model is optimized and trained according to a small-data cartoon image set to obtain an image conversion model with a cartoon style. The style of the input image to be converted is converted according to the image conversion model to obtain a target cartoon image. Through the above method, by modifying the training method of the existing generation network, the encoder can be trained according to the large-data image set and the parameters of the image conversion model can be configured according to the training results. The image conversion model is optimized and trained according to the small-data cartoon image set, so that the small training data can be used to train the generation network to perform style conversion on the image, and the problem that the encoder cannot be applied to the small training data in the training process of the unconditional generation network is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic sub-flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0022] Figure 3 It is another schematic sub-flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0023] Figure 4 It is still another schematic sub-flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0024] Figure 5 It is a subsequent schematic sub-flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0025] Figure 6 It is yet another schematic sub-flowchart of the image conversion method based on a small-data generation network provided by an embodiment of the present invention;

[0026] Figure 7It is a schematic diagram of the subsequent sub - process of the image conversion method based on the small - data generation network provided by the embodiments of the present invention;

[0027] Figure 8 It is a schematic block diagram of the image conversion device based on the small - data generation network provided by the embodiments of the present invention;

[0028] Figure 9 It is a schematic block diagram of the computer device provided by the embodiments of the present invention. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] It should be understood that when used in this specification and the appended claims, the terms "comprises" and "comprising" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0031] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0032] It should be further understood that the term "and / or" used in this specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0033] Please refer to Figure 1 , Figure 1Schematic flowchart of an image conversion method based on a small data generation network provided by an embodiment of the present invention; the image conversion method based on the small data generation network is applied to a user terminal or a management server, and the image conversion method based on the small data generation network is executed through an application software installed in the user terminal or the management server; the user terminal can be used to execute the image conversion method based on the small data generation network to train the generation network and perform style conversion on the input image to be converted by the user to generate a corresponding target cartoon image. The user terminal can be a terminal device such as a desktop computer, a notebook computer, a tablet computer, or a mobile phone. The management server is a server side used to execute the image conversion method based on the small data generation network to train the generation network and obtain the image to be converted uploaded by the user terminal for style conversion to generate a corresponding target cartoon image, such as a server side built within an enterprise or a government department. As Figure 1 shown, the method includes steps S110 to S150.

[0034] S110. Iteratively train the initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model.

[0035] Iteratively train the initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model. A large data image set is pre-stored in the management server or the user terminal, and the initial image conversion model can be iteratively trained according to the large data image set. Specifically, the large data image set contains a large number of real face images, and the sizes of the face images are the same (such as all 1024×1024). Since the acquisition cost of face images is low, a large number of face images can be obtained to form the large data image set. The number of face images contained in the large data image set is usually several thousand, tens of thousands or more. The initial image conversion model can be an intelligent conversion model constructed based on Style GAN (Style-Generative Adversarial Networks). The initial image conversion model contains multiple parameter values, so the initial image conversion model can be iteratively trained according to the face images contained in the large data image set, that is, the parameter values in the model are iteratively adjusted, and the model with the parameter adjustment completed is used as the trained image conversion model.

[0036] In one embodiment, as Figure 2 shown, step S110 includes sub-steps S111, S112, S113, S114, S115, S116, S117, and S118.

[0037] S111. Obtain a face image from the big data image set as the current face image; S112. Extract the input features corresponding to the current face image according to the convolutional input layer of the initial image conversion model, where the input features include hidden variable features and noise features.

[0038] Any face image in the big data image set can be obtained as the current face image, and the initial image convolution model can be trained with the current face image. Specifically, the initial image convolution model includes a convolutional input layer, which can perform convolutional processing on the current face image to obtain corresponding input features. The input features include hidden variable features and noise features. The hidden variable feature z can reflect the hidden feature information of the current face image in the form of a feature vector, and the noise feature B reflects the noise feature information of the current face image in the form of a feature vector. The noise feature B is used to enrich the detail feature information of the generated image.

[0039] S113. Perform feature analysis on the input features according to the multi-layer analysis network in the initial image conversion model to output an output image corresponding to the input features.

[0040] The input features can be analyzed according to the multi-layer analysis layer in the initial image conversion model. Specifically, multiple convolutional processes can be performed based on the hidden variable z to further obtain intermediate hidden variables W. The obtained intermediate hidden variables W are subjected to an affine transformation to obtain feature A. Feature A is a feature vector used to control the overall style of the generated image. There can be multiple variables w (w ∈ W) in the intermediate hidden variables, and each variable w in the intermediate hidden variables corresponds to a feature A. Feature A and the noise feature B are respectively input into each analysis layer of the multi-layer analysis layer. The image sizes output by each analysis layer are different, but the image sizes corresponding to each analysis layer are all multiples of 2, such as 4×4, 8×8, 16×16, etc. The multi-layer basic images obtained through layer-by-layer feature analysis by the multi-layer analysis network are integrated to obtain the corresponding output image. The size of the output image is the same as the size of the current face image (such as 1024×1024).

[0041] S114. Calculate the loss value between the output image and the current face image according to the preset loss function.

[0042] The loss value between the output image and the current face image can be calculated according to a preset loss function. Specifically, the difference value of each pixel point between the output image and the current face image can be obtained. Generally, each pixel point in the output image and the current face image contains pixel values corresponding to three color channels (taking RGB as the reference, including the red, green, and blue color channels). Based on the difference value of each pixel point, the loss value between the two images can be calculated. If the output image is closer to the current face image, the corresponding calculated image loss value is smaller; conversely, the corresponding calculated image loss value is larger.

[0043] For example, the loss function can be expressed by formula (1):

[0044]

[0045] where N is the number of rows of pixel points in the image, M is the number of columns of pixel points in the image, P ij is the pixel value corresponding to the three color channels at the i-th row and j-th column in the output image, and R ij is the pixel value corresponding to the three color channels in the current face image.

[0046] S115. Adjust the parameter values included in the initial image conversion model according to the preset parameter adjustment rule and the loss value.

[0047] The parameter values included in the initial image conversion model can be adjusted based on the parameter adjustment rule and the loss value. Specifically, the parameter adjustment rule can be the gradient descent rule, and the learning rate of gradient descent is configured in the parameter adjustment rule. The updated value of the parameter can be calculated based on the gradient descent rule, the loss value, and the calculated value of each calculation process in the model. The original parameter value of each parameter is updated based on the updated value of the parameter. The parameter values in the model include the parameter values configured in the analysis layer, the convolutional layer, and the convolutional input layer.

[0048] S116. Determine whether there are other unprocessed face images in the big data image set; S117. If there are other unprocessed face images in the big data image set, obtain the next face image as the current face image and return to execute the step of extracting the input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model; S118. If there are no other unprocessed face images in the big data image set, determine the current obtained initial image conversion model as the trained image conversion model.

[0049] It can be determined whether there are other face images in the large - data image set that have not been used for training. If there are, the next face image is continuously obtained as the current face image for model training. If not, the image conversion model obtained from the current training is used as the trained image conversion model. One face image can be used to train the initial image conversion model once. Through another face image, the initial image conversion model after the previous training can be trained again in the same way. Using multiple face images can achieve iterative training of the initial image conversion model.

[0050] In one embodiment, as Figure 3 shown, step S115 includes sub - steps S1151, S1152, and S1153.

[0051] S1151: Determine whether the loss value is not greater than a preset loss threshold; S1152: If the loss value is not greater than the loss threshold, determine the currently obtained initial image conversion model as the trained image conversion model; S1153: If the loss value is greater than the loss threshold, randomly obtain a face image from the large - data image set as the current face image and return to execute extracting the input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model.

[0052] Specifically, after calculating the loss value during the process of training the model, it can also be determined whether the loss value is not greater than a preset loss threshold. If the loss value obtained in a certain calculation is not greater than the loss threshold, it indicates that the currently trained image conversion model can already generate an output image close to the original face image. Then, the training process can be terminated, and the currently obtained image conversion model is used as the trained image conversion model. If the loss value is greater than the loss threshold, it indicates that the model training is insufficient, and the next face image can be continuously obtained to further train the model.

[0053] S120: Train a preset encoder according to the multi - layer analysis network in the image conversion model and the large - data image set to obtain a corresponding target encoder.

[0054] Train a preset encoder according to the multi - layer analysis network in the image conversion model and the large - data image set to obtain a corresponding target encoder. The encoder can be trained according to the image conversion model and the large - data image set. Specifically, the encoder (Enocder) is a virtual structure that can be used to encode images. It can be trained in combination with the multi - layer analysis network in the already trained image conversion model to obtain the trained target encoder.

[0055] In one embodiment, as Figure 4As shown, step S120 includes sub-steps S121, S122, S123, S124, S125, S126, S127, and S128.

[0056] S121. Obtain a face image from the big data image set as the current face image; S122. Perform convolutional processing on the current face image according to the encoder to obtain convolutional feature vectors corresponding to multiple convolutional layers in the encoder.

[0057] A face image can be obtained from the big data image set as the current face image, and convolutional processing is performed on the current face image through an encoder. The encoder includes multiple convolutional layers, and the multiple convolutional layers are arranged in series. The convolutional layer can be used to perform convolutional processing on the pixel values of the pixel points included in the face image. The convolutional result obtained by the previous convolutional layer performing convolutional processing on the image can be used as input information to be input into the next convolutional layer for convolutional processing. Specifically, the convolutional feature vectors obtained by each convolutional layer in the encoder performing convolutional processing on the face image can be obtained. The convolutional feature vector is the convolutional result obtained by the convolutional layer performing convolutional processing, and the convolutional feature vector is composed of a three-dimensional array containing multiple convolutional values. Then, each convolutional layer can correspond to a convolutional feature vector.

[0058] For example, the encoder contains a total of 9 convolutional layers arranged in series. After performing convolutional processing on the initial image through the convolutional processing model, the convolutional feature vectors corresponding to the 9 convolutional layers can be obtained in sequence.

[0059] S123. Input the convolutional feature vectors of the multiple convolutional layers into multiple analysis layers of the multi-layer analysis network respectively to analyze and obtain the corresponding training face image.

[0060] Specifically, the convolutional feature vector in the convolutional layer can be used as the intermediate hidden variable W. A group of convolutional feature vectors can be correspondingly split into multiple variables w, and the multiple variables w together form a group of intermediate hidden variables W. For example, the intermediate hidden variable W = {w1, w2... w 18}, perform an affine transformation on the obtained convolutional feature vector to obtain the feature A. Each intermediate hidden variable corresponds to a feature A, and 18 features A are correspondingly obtained. The feature A is the feature vector used to control the overall style of the generated image. Input the obtained multiple features A into the analysis layers of the multi-layer analysis network respectively. For example, the first two features A are input into the first analysis layer, and the corresponding output image size is 4×4. The third and fourth features A are input into the second analysis layer, and the corresponding output image size is 8×8,... The last two features A are input into the last analysis layer, and the corresponding output image size is 1024×1024. Combine the images output by each analysis layer respectively to obtain the corresponding training face image.

[0061] S124. Calculate the loss value between the training face image and the current face image according to the loss function.

[0062] The loss value between the training face image and the current face image can be calculated according to the loss function. The calculation process of the loss value is the same as the process of obtaining the loss value between the output image and the current face image, which will not be elaborated here.

[0063] S125. Adjust the parameter values included in the encoder according to the parameter adjustment rule and the loss value.

[0064] Parameter values are configured in functional components such as the convolutional layer and affine transformation layer in the encoder. Then, the parameter values included in the encoder can be adjusted according to the parameter adjustment rule and the calculated loss value. The process of adjusting the parameter values is the same as the process of adjusting the parameter values in the initial image conversion model, which will not be elaborated here.

[0065] S126. Determine whether there are other un-trained face images in the big data image set; S127. If there are other un-trained face images in the big data image set, obtain the next face image as the current face image and return to execute the convolutional processing of the current face image according to the encoder to obtain the convolutional feature vectors corresponding to multiple convolutional layers in the encoder respectively; S128. If there are no other un-trained face images in the big data image set, use the currently obtained encoder as the trained target encoder.

[0066] Since it is necessary to train to obtain the latent features of all face images in the big data image set, each face image in the big data image set can be obtained in turn to train the encoder until each face image in the big data image set has completed the training of the encoder, and finally the target encoder is trained.

[0067] S130. Configure the parameters of the image conversion model according to the target encoder to obtain the configured image conversion model.

[0068] Configure the parameters of the image conversion model according to the target encoder to obtain the configured image conversion model. To ensure that the real person photo and the subsequent generated cartoon photo have the same distance and direction in the latent space, the latent space is composed of the above-mentioned hidden variable z, that is, part of the weight layers in the image conversion model. The parameters of the trained image conversion model can be configured through the target encoder, that is, fix part of the weight parameters in the image conversion model. Subsequently, when using the image conversion model to generate cartoon photos, the generated cartoon photos can have the same distance and direction as the real person photos in the latent space.

[0069] In one embodiment, as Figure 5 shown, step S130 includes sub-steps S131 and S132.

[0070] S131. Extract the attribute coding vector corresponding to the big data image set from the target encoder.

[0071] Specifically, the target encoder can be used to encode each face image in the big data image set respectively, so as to extract the corresponding convolution feature vector of each face image. After combining the convolution feature vectors of all face images, they are split into corresponding multiple variables w, and the multiple variables w can be combined into the attribute coding vector corresponding to the big data image set.

[0072] S132. Migrate the attribute coding vector to the image conversion model to configure the intermediate hidden feature parameters in the image conversion model, and obtain the configured image conversion model.

[0073] Migrate the obtained attribute coding vector to the image conversion model, so as to configure the attribute coding vector as the intermediate hidden feature parameter, that is, to fix the corresponding intermediate hidden feature weight parameter in the image conversion model. The attribute coding vector contains multiple variables w, and each variable w corresponds to the feature A required to be input into the multi-layer analysis network in the image conversion model. Then, the attribute coding vector can be directly configured as the input information upstream of the feature A and fixed, and the input information upstream of the feature A remains unchanged during the subsequent use of the image conversion model.

[0074] S140. Optimize and train the configured image conversion model according to the preset training rules and the pre-stored small data cartoon image set, and obtain an image conversion model with a cartoon style.

[0075] Optimize and train the configured image conversion model according to the preset training rules and the pre-stored small data cartoon image set to obtain an image conversion model with a cartoon style. To ensure that the image conversion model can correspondingly generate cartoon-style face images, the configured image conversion model can be optimized and trained according to the training rules and the small data cartoon image set. Among them, the number of cartoon images included in the small data cartoon image set is much smaller than the number of face images in the large data image set. For example, the number of cartoon images included in the small data cartoon image set is between 10 and 100, and the size of each cartoon image is the same as the size of the face image (such as 1024×1024). The weights of some convolutional layers in the image conversion model are fixed, and the weight parameters in other unfixed convolutional layers can be adjusted in this optimization training. After optimizing and training the image conversion model, an image conversion model with a cartoon style can be obtained. Since the image conversion model with a cartoon style has the same distance and direction in the latent space as real photos, the image conversion model with a cartoon style can make the generated cartoon face more like the real human face attributes.

[0076] In one embodiment, as Figure 6 shown, step S140 includes sub-steps S141, S142, S143, S144, S145, S146, S147, and S148.

[0077] S141. Obtain a cartoon image from the small data cartoon image set as the current cartoon image; S142. Extract the cartoon noise feature corresponding to the current cartoon image according to the convolutional input layer in the image conversion model.

[0078] A cartoon image can be obtained as the current cartoon image, and the cartoon noise feature corresponding to the current cartoon image can be extracted. Since this training does not involve the hidden variable part of the image conversion model, only the cartoon noise feature of the cartoon image needs to be obtained.

[0079] S143. Perform feature analysis on the cartoon noise feature according to the multi-layer analysis network in the image conversion model to output a cartoon output image corresponding to the current cartoon image.

[0080] Perform feature analysis on the cartoon noise feature according to the multi-layer analysis network and the fixed intermediate hidden feature weight parameters in the image conversion model, so that the corresponding cartoon output image can be obtained from the multi-layer analysis network. The size of the cartoon output image is the same as the size of the current cartoon image.

[0081] S144. Calculate the detail loss value between the current cartoon image and the cartoon output image according to the detail loss function calculation formula in the training rule; S145. Adjust the parameter value contained in the convolution layer corresponding to the convolution layer identifier in the multi-layer analysis network according to the convolution layer identifier in the training rule and the detail loss value.

[0082] Specifically, the detail loss function can be calculated based on the detail features of the image. The detail features include face shape, eye features, hairstyle, etc. The facial contours of the current cartoon image and the cartoon output image can be obtained respectively, and the facial length, facial width, facial area, etc. can be calculated to reflect the feature information of the face shape. The pupil color, pupil area, eye length, eye area, etc. of the two images can be obtained respectively to reflect the feature information of the eye features. The hair area, the ratio of hair area to facial area, etc. can be obtained respectively to reflect the feature information of the hairstyle features. The detail loss value of the two images between the detail features is calculated according to the detail loss function. The training rules also include convolutional layer identifiers. The convolutional layer identifiers are information used to identify the convolutional layers whose parameters can be adjusted. For example, if the convolutional layer identifiers are 5, 6, 7, 8, and 9, when adjusting the parameter values in the multi-layer analysis network, only the parameter values contained in the fifth, sixth, seventh, eighth, and ninth convolutional layers are adjusted. The parameter values of the other convolutional layers are locked and cannot be adjusted. The specific process of adjusting the parameter values in the corresponding convolutional layer is the same as the process of adjusting the parameter values in the initial image conversion model, and will not be repeated here.

[0083] S146. Determine whether the small data cartoon image set contains other untrained cartoon images; S147. If the small data cartoon image set contains other untrained face images, obtain the next cartoon face image as the current cartoon image and return to execute the cartoon noise feature corresponding to the current cartoon image extracted according to the convolution input layer in the image conversion model; S148. If the small data cartoon image set does not contain other untrained cartoon images, determine the currently obtained image conversion model as the trained image conversion model containing a cartoon style.

[0084] After adjusting the parameter values in the multi-layer analysis network once, it can be determined whether the small data cartoon image set contains other untrained cartoon images. If so, other cartoon images are continuously obtained to optimize the image conversion model. If not, the currently obtained image conversion model is determined as the trained image conversion model containing the cartoon style.

[0085] S150 : If an input image to be converted is received, style conversion is performed on the image to be converted according to the image conversion model containing a cartoon style to obtain a target cartoon image corresponding to the image to be converted.

[0086] If the input image to be converted is received, the size of the image to be converted can be the same as the size of the face image (e.g., both are 1024×1024). According to the image conversion model including the cartoon style, the style conversion is performed on the image to be converted to obtain a target cartoon image corresponding to the image to be converted. The user can input the image to be converted, and if the image to be converted is a real-person photo, the style conversion can be performed on the image to be converted through the optimized and trained image conversion model, so as to obtain a target cartoon image corresponding to the image to be converted. The target cartoon image is an image in cartoon style, and the target cartoon image is more inclined to the attributes of the real-person face. Therefore, the target cartoon image is a cartoon-style image biased towards the attributes of the real-person face.

[0087] In one embodiment, as Figure 7 shown, step S150 includes sub-steps S151 and S152.

[0088] S151. Extract the image noise feature corresponding to the image to be converted according to the convolutional input layer in the image conversion model; S152. Perform feature analysis on the image noise feature according to the multi-layer analysis network in the image conversion model to convert and obtain a target cartoon image corresponding to the image to be converted.

[0089] The image noise feature of the image to be converted can be extracted according to the convolutional input layer in the image conversion model, and the feature analysis is performed on the image noise feature according to the multi-layer analysis network and the fixed intermediate hidden feature weight parameters in the image conversion model, so that the corresponding target cartoon image can be obtained from the multi-layer analysis network. The size of the target cartoon image is the same as the size of the image to be converted.

[0090] In the image conversion method based on the small-data generation network provided in the embodiment of the present invention, the initial image conversion model is iteratively trained according to the large-data image set to obtain a trained image conversion model. The encoder is trained according to the large-data image set to obtain a target encoder and the parameters of the image conversion model are configured. Then, the configured image conversion model is optimized and trained according to the small-data cartoon image set to obtain an image conversion model including the cartoon style. The style conversion is performed on the input image to be converted according to the image conversion model to obtain a target cartoon image. Through the above method, by modifying the training method of the existing generation network, the encoder can be trained based on the large-data image set and the parameters of the image conversion model can be configured according to the training results. The image conversion model is optimized and trained according to the small-data cartoon image set, so that the small training data can be used to train the generation network to perform style conversion on the image, and the problem that the encoder cannot be applied to the small training data in the training process of the unconditional generation network is solved.

[0091] An embodiment of the present invention further provides an image conversion device based on a small data generation network. The image conversion device based on the small data generation network can be configured in a user terminal or a management server, and is used to execute any one of the foregoing embodiments of the image conversion method based on the small data generation network. Specifically, please refer to Figure 8 , Figure 8 which is a schematic block diagram of the image conversion device based on the small data generation network provided by the embodiment of the present invention.

[0092] As Figure 8 shown, the image conversion device 100 based on the small data generation network includes an initial image conversion model training unit 110, an encoder training unit 120, a parameter configuration unit 130, a model optimization training unit 140, and a target cartoon image acquisition unit 150.

[0093] The initial image conversion model training unit 110 is configured to iteratively train the initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model.

[0094] In a specific embodiment, the initial image conversion model training unit 110 includes sub-units: a current face image acquisition unit, configured to acquire a face image in the large data image set as the current face image; an input feature acquisition unit, configured to extract an input feature corresponding to the current face image according to a convolutional input layer of the initial image conversion model, where the input feature includes a hidden variable feature and a noise feature; an output image acquisition unit, configured to perform feature analysis on the input feature according to a multi-layer analysis network in the initial image conversion model to output an output image corresponding to the input feature; a loss value calculation unit, configured to calculate a loss value between the output image and the current face image according to a preset loss function; a parameter value adjustment unit, configured to adjust parameter values included in the initial image conversion model according to a preset parameter adjustment rule and the loss value; an image set judgment unit, configured to judge whether there are other face images in the large data image set that have not been trained; a return execution unit, configured to, if there are other face images in the large data image set that have not been trained, acquire the next face image as the current face image and return to execute extracting an input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model; an image conversion model determination unit, configured to, if there are no other face images in the large data image set that have not been trained, determine the current initial image conversion model as the trained image conversion model.

[0095] In a specific embodiment, the initial image conversion model training unit 110 further includes sub-units: a loss value judgment unit, configured to judge whether the loss value is not greater than a preset loss threshold; a determination unit, configured to, if the loss value is not greater than the loss threshold, determine the currently obtained initial image conversion model as the trained image conversion model; and a return unit, configured to, if the loss value is greater than the loss threshold, randomly obtain a face image from the big data image set as the current face image and return to execute the extraction of the input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model.

[0096] An encoder training unit 120, configured to train a preset encoder according to the multi-layer analysis network in the image conversion model and the big data image set, so as to obtain a corresponding target encoder.

[0097] In a specific embodiment, the encoder training unit 120 includes sub-units: a face image acquisition unit, configured to acquire a face image from the big data image set as the current face image; a convolutional feature vector acquisition unit, configured to perform convolutional processing on the current face image according to the encoder, so as to obtain convolutional feature vectors corresponding to multiple convolutional layers in the encoder; a feature analysis unit, configured to respectively input the feature maps of multiple convolutional layers into multiple analysis layers of the multi-layer analysis network, so as to analyze and obtain corresponding training face images; an image loss value calculation unit, configured to calculate the loss value between the training face image and the current face image according to the loss function; an encoder parameter value adjustment unit, configured to adjust the parameter values included in the encoder according to the parameter adjustment rule and the loss value; a judgment unit, configured to judge whether there are other face images in the big data image set that have not been trained; and a return processing unit, configured to, if there are other face images in the big data image set that have not been trained, acquire the next face image as the current face image and return to execute the convolutional processing of the current face image according to the encoder, so as to obtain convolutional feature vectors corresponding to multiple convolutional layers in the encoder; and a target encoder acquisition unit, configured to, if there are no other face images in the big data image set that have not been trained, determine the currently obtained encoder as the trained target encoder.

[0098] A parameter configuration unit 130, configured to perform parameter configuration on the image conversion model according to the target encoder, so as to obtain the configured image conversion model.

[0099] In a specific embodiment, the parameter configuration unit 130 includes subunits: an attribute coding vector acquisition unit, configured to extract an attribute coding vector corresponding to the big data image set from the target encoder; and an intermediate hidden feature parameter configuration unit, configured to migrate the attribute coding vector to the image conversion model to configure intermediate hidden feature parameters in the image conversion model, thereby obtaining the configured image conversion model.

[0100] The model optimization training unit 140 is configured to optimize and train the configured image conversion model according to a preset training rule and a pre-stored small data cartoon image set, thereby obtaining an image conversion model with a cartoon style.

[0101] In a specific embodiment, the model optimization training unit 140 includes subunits: a current cartoon image acquisition unit, configured to acquire a cartoon image from the small data cartoon image set as the current cartoon image; a cartoon noise feature acquisition unit, configured to extract a cartoon noise feature corresponding to the current cartoon image according to a convolutional input layer in the image conversion model; a cartoon output image acquisition unit, configured to perform feature analysis on the cartoon noise feature according to a multi-layer analysis network in the image conversion model to output a cartoon output image corresponding to the current cartoon image; a detail loss value calculation unit, configured to calculate a detail loss value between the current cartoon image and the cartoon output image according to a detail loss function calculation formula in the training rule; a convolutional layer parameter value adjustment unit, configured to adjust parameter values included in a convolutional layer corresponding to the convolutional layer identifier in the multi-layer analysis network according to the convolutional layer identifier and the detail loss value in the training rule; a cartoon image set determination unit, configured to determine whether other untrained cartoon images are included in the small data cartoon image set; a return extraction unit, configured to, if other untrained face images are included in the small data cartoon image set, acquire the next cartoon face image as the current cartoon image and return to execute the step of extracting a cartoon noise feature corresponding to the current cartoon image according to the convolutional input layer in the image conversion model; and a conversion model determination unit, configured to, if no other untrained cartoon images are included in the small data cartoon image set, determine the currently obtained image conversion model as the trained image conversion model with a cartoon style.

[0102] The target cartoon image acquisition unit 150 is configured to, if a to-be-converted image is received, perform style conversion on the to-be-converted image according to the image conversion model with a cartoon style, thereby obtaining a target cartoon image corresponding to the to-be-converted image.

[0103] In a specific embodiment, the target cartoon image acquisition unit 150 includes subunits: an image noise feature acquisition unit, configured to extract, according to a convolutional input layer in the image conversion model, an image noise feature corresponding to the image to be converted; and an image conversion unit, configured to perform feature analysis on the image noise feature according to a multi-layer analysis network in the image conversion model, so as to convert and obtain a target cartoon image corresponding to the image to be converted.

[0104] The image conversion device based on the small-data generation network provided in the embodiments of the present invention applies the above-mentioned image conversion method based on the small-data generation network, iteratively trains an initial image conversion model according to a large-data image set to obtain a trained image conversion model, trains an encoder according to the large-data image set to obtain a target encoder and configures parameters for the image conversion model, optimally trains the configured image conversion model according to a small-data cartoon image set to obtain an image conversion model including a cartoon style, and performs style conversion on an input image to be converted according to the image conversion model to obtain a target cartoon image. Through the above method, by modifying the training method of the existing generation network, the encoder can be trained based on a large-data image set and the parameters of the image conversion model can be configured according to the training result, and the image conversion model can be optimally trained according to a small-data cartoon image set, so that the generation network can be trained with small training data to perform style conversion on an image, and the problem that the encoder cannot be applied to small training data in the training process of the unconditional generation network is solved.

[0105] The above-mentioned image conversion device based on the small-data generation network can be implemented in the form of a computer program, and the computer program can run on a computer device as shown in Figure 9 shown.

[0106] Please refer to Figure 9 , Figure 9 which is a schematic block diagram of the computer device provided in the embodiments of the present invention. The computer device can be a user terminal or a management server for executing the image conversion method based on the small-data generation network to train a generation network and perform style conversion on an image.

[0107] Refer to Figure 9 , the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501. Among them, the memory can include a storage medium 503 and an internal memory 504.

[0108] The storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can be enabled to execute the image conversion method based on the small-data generation network. Among them, the storage medium 503 can be a volatile storage medium or a non-volatile storage medium.

[0109] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0110] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can be caused to execute an image conversion method based on a small data generation network.

[0111] The network interface 505 is used for network communication, such as providing the transmission of data information, etc. Those skilled in the art can understand that Figure 9 the structure shown in is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0112] Among them, the processor 502 is used to run the computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned image conversion method based on a small data generation network.

[0113] Those skilled in the art can understand that Figure 9 the embodiments of the computer device shown in do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structures and functions of the memory and the processor are the same as those of Figure 9 the embodiments shown, and will not be described in detail here.

[0114] It should be understood that in the embodiments of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0115] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps included in the above-mentioned image conversion method based on small data generation network are implemented.

[0116] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0117] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. Units with the same function can also be aggregated into one unit. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be electrical, mechanical, or other forms of connection.

[0118] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.

[0119] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0120] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned computer-readable storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs that can store program codes.

[0121] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image conversion method based on a small data generation network, characterized in that, The method includes: Iteratively training an initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model; Training a pre-set encoder according to a multi-layer analysis network in the image conversion model and the large data image set to obtain a corresponding target encoder; Configuring parameters of the image conversion model according to the target encoder to obtain the configured image conversion model; Optimally training the configured image conversion model according to a pre-set training rule and a pre-stored small data cartoon image set to obtain an image conversion model including a cartoon style; If a to-be-converted image is received, performing style conversion on the to-be-converted image according to the image conversion model including the cartoon style to obtain a target cartoon image corresponding to the to-be-converted image; The iteratively training an initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model includes: Obtaining a face image in the large data image set as a current face image; Extracting an input feature corresponding to the current face image according to a convolutional input layer of the initial image conversion model, where the input feature includes a hidden variable feature and a noise feature; the hidden variable feature is a hidden feature information of the current face image presented in the form of a feature vector, and the noise feature is a noise feature information of the current face image presented in the form of a feature vector, and the noise feature is used to enrich detailed feature information of a generated image; Performing feature analysis on the input feature according to a multi-layer analysis network in the initial image conversion model to output an output image corresponding to the input feature; Calculating a loss value between the output image and the current face image according to a pre-set loss function; Adjusting parameter values included in the initial image conversion model according to a pre-set parameter adjustment rule and the loss value; Judging whether there are other face images in the large data image set that have not been trained; If there are other face images in the large data image set that have not been trained, obtaining the next face image as the current face image and returning to execute extracting the input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model; If there are no other face images in the large data image set that have not been trained, determining the current obtained initial image conversion model as the trained image conversion model.

2. The image conversion method based on a small data generation network according to claim 1, characterized in that After adjusting the parameter values included in the initial image conversion model according to the pre-set parameter adjustment rule and the loss value, it further includes: Judging whether the loss value is not greater than a pre-set loss threshold; If the loss value is not greater than the loss threshold, determining the current obtained initial image conversion model as the trained image conversion model; If the loss value is greater than the loss threshold, randomly obtaining a face image in the large data image set as the current face image and returning to execute extracting the input feature corresponding to the current face image according to the convolutional input layer of the initial image conversion model.

3. The image conversion method based on a small data generation network according to claim 1, wherein Training a pre - set encoder according to the multi - layer analysis network in the image conversion model and the large - data image set to obtain a corresponding target encoder, including: Obtaining a face image in the large - data image set as the current face image; Performing convolution processing on the current face image according to the encoder to obtain convolution feature vectors corresponding to multiple convolutional layers in the encoder; Inputting the feature maps of multiple convolutional layers into multiple analysis layers of the multi - layer analysis network respectively to analyze and obtain corresponding training face images; Calculating the loss value between the training face image and the current face image according to the loss function; Adjusting the parameter values included in the encoder according to the parameter adjustment rule and the loss value; Judging whether there are other un - trained face images in the large - data image set; If there are other un - trained face images in the large - data image set, obtaining the next face image as the current face image and returning to execute performing convolution processing on the current face image according to the encoder to obtain convolution feature vectors corresponding to multiple convolutional layers in the encoder; If there are no other un - trained face images in the large - data image set, taking the currently obtained encoder as the trained target encoder.

4. The image conversion method based on a small data generation network according to claim 1, wherein Configuring the parameters of the image conversion model according to the target encoder to obtain the configured image conversion model, including: Extracting an attribute encoding vector corresponding to the large - data image set from the target encoder; Transferring the attribute encoding vector to the image conversion model to configure the intermediate hidden feature parameters in the image conversion model to obtain the configured image conversion model.

5. The image conversion method based on a small data generation network according to claim 1, wherein Optimally training the configured image conversion model according to the pre - set training rule and the pre - stored small - data cartoon image set to obtain an image conversion model with a cartoon style, including: Obtaining a cartoon image in the small - data cartoon image set as the current cartoon image; Extracting a cartoon noise feature corresponding to the current cartoon image according to the convolutional input layer in the image conversion model; Performing feature analysis on the cartoon noise feature according to the multi - layer analysis network in the image conversion model to output a cartoon output image corresponding to the current cartoon image; Calculating the detail loss value between the current cartoon image and the cartoon output image according to the detail loss function calculation formula in the training rule; Adjusting the parameter values included in the convolutional layer corresponding to the convolutional layer identifier in the multi - layer analysis network according to the convolutional layer identifier in the training rule and the detail loss value; Judging whether there are other un - trained cartoon images in the small - data cartoon image set; If there are other un - trained face images in the small - data cartoon image set, obtaining the next cartoon face image as the current cartoon image and returning to execute extracting a cartoon noise feature corresponding to the current cartoon image according to the convolutional input layer in the image conversion model; If there are no other untrained cartoon images in the small data cartoon image set, determine the currently obtained image conversion model as the trained image conversion model with a cartoon style.

6. The image conversion method based on a small data generation network according to claim 1, wherein Performing style conversion on the image to be converted according to the image conversion model with a cartoon style to obtain a target cartoon image corresponding to the image to be converted includes: Extracting an image noise feature corresponding to the image to be converted according to the convolutional input layer in the image conversion model; Performing feature analysis on the image noise feature according to the multi-layer analysis network in the image conversion model to convert and obtain a target cartoon image corresponding to the image to be converted.

7. An image conversion device based on a small data generation network, characterized in that, The device is used to execute the image conversion method based on a small data generation network according to any one of claims 1-6. The device includes: An initial image conversion model training unit, configured to iteratively train an initial image conversion model according to a pre-stored large data image set to obtain a trained image conversion model; An encoder training unit, configured to train a pre-set encoder according to the multi-layer analysis network in the image conversion model and the large data image set to obtain a corresponding target encoder; A parameter configuration unit, configured to perform parameter configuration on the image conversion model according to the target encoder to obtain the configured image conversion model; A model optimization training unit, configured to perform optimization training on the configured image conversion model according to a pre-set training rule and a pre-stored small data cartoon image set to obtain an image conversion model with a cartoon style; A target cartoon image acquisition unit, configured to, if a received image to be converted is received, perform style conversion on the image to be converted according to the image conversion model with a cartoon style to obtain a target cartoon image corresponding to the image to be converted.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image conversion method based on a small data generation network according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the image conversion method based on a small data generation network according to any one of claims 1 to 7.

Images