Method for transferring features of a first image to a second image

Through a neural generator network with dual discriminator structure, specific features are steadily copied from images to another image, solving the complexity of feature transfer under data set imbalance, generating clear pseudo-feature images, improving the robustness of the model and data enhancement capabilities.

CN111753980BActive Publication Date: 2025-07-25ROBERT BOSCH GMBH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010220100.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-03-26
Filing Date
2020-03-25
Publication Date
2025-07-25
Estimated Expiration
2040-03-25

AI Technical Summary

Technical Problem

The prior art is difficult to effectively copy specific features from one image to another while keeping other areas of the image unchanged, especially in the case of unbalanced data sets, where learning is complex and unstable.

Method used

A neural generator network adopts a dual discriminator structure, and alternately trains the generator and discriminator networks, uses feature images and pure images to generate intermediate generator images, and realizes feature transfer through superposition. Use loss functions to control the training process to ensure the stability and accuracy of the generator network.

Benefits of technology

It realizes the stable copying of specific features from the image to another image under the unbalanced data set, and generates clear pseudo-feature images, improving the robustness of the model and data enhancement capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111753980B_ABST
    Figure CN111753980B_ABST
Patent Text Reader

Abstract

A method for transferring features of a first image to a second image includes: generating an intermediate generator image by providing a feature image among a plurality of feature images including the features and a pure image without the feature among a plurality of pure images without the feature to a generator network; superimposing the intermediate generator image with the feature image to construct a pseudo-image; providing the pseudo-image and the pure image to a first discriminator network; calculating a loss function of the first discriminator network in any odd training sequence; superimposing the inverted intermediate generator image with the pure image to construct a pseudo-feature image; providing the pseudo-feature image and the feature image to a second discriminator network; calculating a loss function of the second discriminator network in any odd training sequence; recalculating network parameters of the first discriminator network and the second discriminator network in any odd training sequence; calculating a loss function of the generator network in any even training sequence; and recalculating network parameters of the generator network in any even training sequence.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for transferring features of a first image to a second image using an artificial neural network, and a method for training these artificial neural networks. Background Art

[0002] Due to the success of deep generative adversarial networks (GANs), image generation has attracted widespread interest. These models learn the distribution of features in a real dataset and use the trained model to generate images. Although generating images from a random vector or image class suffers from blurry images, image-to-image generation models are able to generate clear and realistic images. Different methods have been proposed to solve the hard image generation problem. Most of these models learn the distribution of the dataset in the image space and sample the complete image from the trained model conditioned on the image.

[0003] Generating images from given features has been studied using deep feature interpolation in a trained model that uses the given features to generate images or manipulate features in the GAN framework. A two-stage scheme has been applied to train the entire framework and use the given features to generate images. To ensure the realistic appearance of the generated images, an algorithm including two networks - a transformation network and an enhancement network - has been used. Such algorithms typically attempt to learn the implicit distribution of the attributes.

[0004] For some practical applications of image classification, such as defect detection in production automation and disease detection in medical image classification, the dataset of images is highly imbalanced. Compared with images without anomalies, images showing anomalies (e.g., product images with defects, medical images with malignant tumor cells) are much fewer. Based on such a dataset, training a robust image classifier becomes challenging. Summary of the Invention

[0005] Accordingly, the present invention relates to a method for transferring the features of a feature image including features to a pure image using an artificial neural generator network as described in the independent claims, and a method for training an artificial neural generator network using the features. The present invention also relates to a computer program product and a computer-readable storage medium configured to implement these methods.

[0006] Advantageous modifications of the present invention are stated in the dependent claims. All combinations of at least two of the features disclosed in the description, claims, and drawings fall within the scope of the present invention. To avoid repetition, the features disclosed according to the method should also apply to the mentioned systems and devices and are claimable according to the mentioned systems and devices.

[0007] The idea underlying the present invention is to avoid teaching the distribution of features within an image to a neural network, but rather to teach to copy only specific features from an image to another image while keeping other areas of the image unchanged. Otherwise, since some features (e.g., scratches on a product) are rather random, the implicit distribution of these features will complicate learning due to the need for a large amount of data.

[0008] To achieve these and other advantages and in accordance with the purpose of the present invention, as embodied and broadly described herein, there is provided a method for training an artificial neural generator network that transfers the feature of a feature image including a feature to a pure image by repeating a training sequence of the generator network until the absolute value of the loss function of the generator network is lower than a predefined threshold.

[0009] The neural generator network can be an artificial neural network (ANN), which is a computing system vaguely inspired by the biological neural networks that make up animal brains. A neural network itself is not an algorithm, but a framework in which many different machine learning algorithms work together and process complex data inputs. Such a system typically learns to perform tasks by considering examples without being programmed with any task-specific rules. An ANN is based on a collection of connected units or nodes called artificial neurons. Each connection can transmit a signal from one artificial neuron to another. The artificial neuron receiving the signal can process the signal and then signal additional artificial neurons to which it is connected.

[0010] In a common ANN implementation, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is computed by some non-linear function of the sum of its inputs. The weights of the artificial neurons and the edges are typically adjusted as learning proceeds. The weights increase or decrease the strength of the signal at the connection. An artificial neuron can have a threshold such that it sends a signal only when the aggregated signal crosses that threshold. Typically, artificial neurons are aggregated into layers. Different layers can perform different kinds of transformations on their inputs. Signals travel from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0011] The aforementioned training sequence includes the following steps: generating an intermediate generator image by providing the generator network with a feature image including the feature among a plurality of feature images including the feature and a pure image without the feature among a plurality of pure images without the feature.

[0012] When performing the training sequence of the method as described above, a plurality of feature images including the feature to be transferred to a plurality of pure images can be used as a set of training images for the generator network, and the plurality of pure images do not have the feature.

[0013] The generator network generates an intermediate generator image at the output of the generator network, which is produced from the image provided at the input of the generator network. The generator network is specified by its network parameters, which are used for a specific task of the generator network and, together with the input image, determine the output of the generator network.

[0014] Another step of the training sequence is to superimpose the intermediate generator image with a feature image including the feature to construct a pseudo-image. Due to the structure of the neural network framework, the intermediate generator image is a negative image of the feature. Depending on the performance quality of the generator network, the superimposition of the feature image and the intermediate generator image may result in the generated pseudo-image no longer including the feature.

[0015] Another step of the training sequence is to provide the pseudo-image and the pure image to the first discriminator network.

[0016] Another step is to compute the loss function of the first discriminator network in any odd-numbered training sequence.

[0017] Due to this data flow and structure, the first discriminator network has the task of predicting whether the pseudo-image is a real image.

[0018] A discriminator is typically composed of a sequence of layers. The first layer of the discriminator receives an image, and the discriminator generates some values that indicate the probability that the input image is real or pseudo. For both the generator and the discriminator, typical layers are: convolutional network layers (e.g., with or without dilation, with or without depthwise separation), deconvolutional layers, pooling layers (e.g., max pooling, average pooling), normalization layers (e.g., batch normalization, layer normalization, local response normalization, and instance normalization), activation functions (e.g., rectified linear unit (ReLU), leaky rectified linear unit (leaky ReLU), exponential linear unit (elu), scaled exponential linear unit (selu), sigmoid function, or tanh function), ResNet, or a block of ResNet.

[0019] The training sequence further involves the following step: superimposing the previously inverted intermediate generator image with the pure image to construct a pseudo-feature image. Since the intermediate generator image is a negative image of the feature as discussed above, the intermediate generator image must first be inverted to add the feature to the image that initially did not have the feature by superimposing the feature with the image. This results in a pseudo-image that includes the feature and is based on the pure image.

[0020] Providing the pseudo-feature image and the feature image to the second discriminator network is another step of the training sequence. In this way, as a next step, it is possible to compute the loss function of the second discriminator network in any odd-numbered training sequence.

[0021] As a further step of the described method, the network parameters of the first and second discriminator networks are recalculated for any odd training sequence. Alternatively, this recalculation of the respective network parameters for each network is performed using the generator network for any other sequence.

[0022] Count the training sequences, and if the count is odd, perform the step of recalculating the network parameters of the first and second discriminator networks, otherwise do not perform the update calculation. This update of the parameters of the generator network and the discriminator network can be performed by a method called backpropagation.

[0023] Another step of the training sequence is to calculate the loss function of the generator network for any even training sequence. The loss function can be a measure of the quality of the generator network for implementing the desired transfer of features.

[0024] The absolute value of the loss function of the generator network determines the end of the training sequence. If the absolute value of the loss function is below a predefined threshold, the training sequence of the generator network terminates, and the generator network can be called a trained generator network.

[0025] In any even-numbered training sequence of the generator network, the network parameters of the generator network are recalculated. This means that the recalculations of the generator network and the two discriminator networks are performed alternately.

[0026] An improvement of using the described method is that training the generator network and recalculating its parameters in the described manner using two discriminator networks results in a very stable training behavior of the generator network for different images and features to be transferred. Careful hyperparameter tuning is not required. Using only one discriminator network is less stable.

[0027] Another benefit of the disclosed method is the interpretability of the intermediate results, because in addition to generating the required pseudo-feature images, the generator network also generates intermediate generator images that represent the features to be transferred, such as defects in a product. This gives an intuitive idea about the additional pseudo-feature images.

[0028] According to one aspect of the present invention, the calculation of the loss function of the generator takes into account the outputs of the first and second discriminators.

[0029] According to another aspect of the present invention, the loss function of the generator is calculated by using the following formula:

[0030]

[0031] where Lg represents the loss function of the generator network, E x1〜p(x) is the expectation of the distribution of real images without attributes, and E x2〜p(x+)is the expectation of the true image distribution with the property. x1 is a sample in the image space, D1(x1) is the output of the discriminator, and G is the output of the generator.

[0032] According to another aspect of the described method, the artificial neural generator network for the described method is a convolutional encoder-decoder neural network.

[0033] Additionally, a method is provided for transferring the feature of a feature image including the feature to a receiving image by means of an artificial neural generator network, which is trained according to the described method for training the artificial neural generator network.

[0034] A method is disclosed for transferring the feature of a feature image to a receiving image by means of an artificial neural generator network, wherein the generator network is trained according to the above method for training the artificial neural generator network.

[0035] The steps of the method for transferring the feature are to provide to the input of the artificial neural generator network a feature image including the feature and a receiving image without the feature (the receiving image should be the basis for the pseudo-feature image), wherein the generator network provides an intermediate generator image at the output of the generator network.

[0036] The next step of the method for transferring the feature is to superimpose the inverted intermediate generator image with the receiving image to transfer the feature of the feature image to the receiving image. This results in a pseudo-feature image including the feature and based on the receiving image.

[0037] The technical application of the method for transferring the feature from a feature image including the feature to another image can be used for data augmentation. If the training data set for training the neural network is small, the described method can be used to generate more training images for improved model training. Since the number of non-defective images is not limited and the required number of images including the feature to be transferred is small, this generates a large set of real-appearance images with anomalies. As an example value, it can be used for hundreds of images including the required feature. The described method does not require a large number of defective images to learn how to apply the feature to images of non-defective products. This is because it is not necessary to learn the feature distribution in the images.

[0038] For example, such data augmentation methods can support automated optical inspection. When the training data set is small, since there are only a small number of images from a small number of defects during production, this method can be used to generate more training images for better model training. With this improved model, for example, defect detection based on a detection method through a neural network (e.g., for optical inspection) can be improved. For example, if at least some defective product samples are available, product images with defects such as scratches can be generated based on product images without defects.

[0039] Another technical application can be image analysis for medical use. Since the disclosed method generates intermediate generator images representing features that can be related to features indicating characteristic tissue changes modified based on tumors. This differentiation helps to divide the image into benign pixels and tumor pixels.

[0040] An additional technical application of the disclosed method can be to apply the method as, for example, a mobile phone application to enable modifying a person's image with sunglasses worn by another person and performing the modification when an image of the sunglasses is available.

[0041] Additionally, a method for generating a pseudo-feature image including a feature is described. The steps of the method are to iteratively implement the method for transferring the features of a feature image according to the above, where a feature image including the feature is selected from a first plurality of feature images and repeatedly provided to a generator network, while each received image selected from a second plurality of received images is uniquely provided to the generator network. The number of elements in the second plurality is higher than the number of elements in the first plurality.

[0042] If the number of original images including such features is limited, this enables the user to artificially generate multiple images including the desired features by transferring the features of the feature images to another image.

[0043] A computer program product including instructions is described, which when executed by a computer cause the computer to implement the method according to claims 1 to 4.

[0044] A computer-readable storage medium including instructions is described, which when executed by a computer cause the computer to implement the method according to claims 1 to 4.

[0045] A system is described, which includes an artificial neural network system executed by one or more computers, and the artificial neural network system is configured to implement the method according to claims 1 to 4. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated in and constitute a part of this application. The drawings illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. In the drawings:

[0047] Figure 1 illustrates the data flow within the framework of a neural network, the neural network including a generator and two discriminators;

[0048] Figure 2 schematically shows the structure of an artificial neural generator network;

[0049] Figure 3 schematically shows the structure of an artificial neural discriminator network;

[0050] Figure 4 shows the steps of a method for training the generator network; and

[0051] Figure 5 shows the data flow for transferring the features of a feature image to a receiving image. Detailed Description

[0052] Figure 1 Illustrates the data flow of the framework of a neural network, the neural network including a generator network (G) 1, a first discriminator network (D1) 2, and a second discriminator network (D2) 3, for training an artificial neural generator - network (G) 1 to transfer the features included in a feature image to a pure image. This framework can be used to enable the generator network (G) 1 to learn to transfer features from a feature image I+ to a pure image I, which should receive the features, for example because it does not include the features.

[0053] The specifications of the features to be transferred can depend on the specific task. For example, for facial image analysis, the features can be "beard", "glasses", or "smile". For optical inspection during production, the feature can be "defect".

[0054] A feature image I+ (i.e., an image including the features to be transferred) and a pure image I (i.e., an image without this feature) are provided at the input 1a or 1b of the generator network (G) 1. The input data of the generator network (G) 1 can be a stack of two images, and the output of the generator network (G) 1 can be an image having the same resolution as the input images.

[0055] As Figure 1As indicated, the generator output 1c can provide an intermediate generator image as the result of its input and the generator network (G) 1, and can be superimposed with the feature image I+ and provided to the first discriminator network D1 at the first input 2a. The second input 2b of the discriminator (D1) 2 is provided with the pure image I. The discriminator (D1) 2 provides its discrimination result at its output 2c.

[0056] In Figure 1 it is also indicated that the intermediate generator image is inverted, superimposed with the pure image I, and provided to the first input 3a of the second discriminator network (D2) 3. The second input 3b of the second discriminator network (D2) 3 is provided with the feature image I+. The discriminator (D2) 3 provides its discrimination result at its output 3c.

[0057] As Figure 1 shown, the entire data flow results in the generation of two pseudo-images, one pseudo-image having features obtained from two real images and the other not, with the feature image and the pure image being provided to the generator.

[0058] In addition to the fact that this framework of the neural network involves the generator network and the discriminator framework, it is also distinguished from the generative adversarial network (GAN) by including the second discriminator D2, and provides two images to the generator network G instead of noise in the case of the GAN network. Additionally, the intermediate generator image provided at the output 1c of the generator network G is not directly passed to the discriminator, but is added to one of the two input images before they are provided to one of the discriminators.

[0059] The architecture of the generator network can be any architecture configured to receive a stack of two images at its input stage and provide an image at the output stage, such as an autoencoder or any architecture for semantic segmentation. An embodiment of the generator network is discussed below with reference to Figure 2 discussed.

[0060] In Figure 2 it is schematically shown the structure of an artificial encoder-decoder neural network. The architecture of such an encoder-decoder neural network generally consists of two parts. The first part is a sequence of layers that downsample the input grid to a lower resolution, with the aim of retaining the desired information and dumping redundant information. The second part is a sequence of layers that upsample the output of the first part and restore the desired output resolution (e.g., the input resolution). Optionally, there may be additional skip connections directly connecting certain layers in the first part and the second part.

[0061] An embodiment of the generator network (G) 2 can have the architecture of an encoder-decoder network 20, which can be used to generate an output tensor representing an intermediate generator image. The encoder-decoder network 20 can consist of a convolutional neural network (CNN) encoder 23 and a CNN decoder 24. Layer 21 is the input layer 22 of the encoder-decoder network 20, and layer 22 is the output layer of the encoder-decoder network 20.

[0062] The encoder 23 of the encoder-decoder network 20 is constructed from d (e.g., d = 8) blocks. Each block can contain a layer or a sequence of layers or even contain other layer blocks. There can be a convolutional layer with a kernel number of N1 (e.g., N1 = 32) and a stride of 2; a leaky ReLU layer, a convolutional layer with a kernel number of N2 (e.g., N2 = 64) and a stride of 2, a normalization layer (e.g., instance normalization); a leaky ReLU layer, a convolutional layer with a kernel number of N3 (e.g., N3 = 128) and a stride of 2, a normalization layer (e.g., instance normalization); multiple layer blocks, each block containing a leaky ReLU layer, a convolutional layer with a kernel number of N4 (e.g., N4 = 256) and a stride of 2, a normalization layer (e.g., instance normalization).

[0063] The decoder 24 of the encoder-decoder network 20 is also constructed from d blocks. The decoder can contain multiple layer blocks, each block can contain a leaky ReLU layer, a transposed convolutional layer with a kernel number of N5 (e.g., N5 = 256) and a stride of 2, a normalization layer (e.g., instance normalization); a leaky ReLU layer, a transposed convolutional layer with a kernel number of N6 (e.g., N6 = 128) and a stride of 2, a normalization layer; a leaky ReLU layer, a transposed convolutional layer with a kernel number of N7 (e.g., N7 = 128) and a stride of 2, a normalization layer; a leaky ReLU layer, a transposed convolutional layer with a kernel number of N8 (e.g., N8 = 64) and a stride of 2, a normalization layer; and a leaky ReLU layer, a transposed convolutional layer with a kernel number of N9 (e.g., N9 = 32) and a stride of 2, a tanh layer. Table 1a describes these layers in more detail.

[0064] Table 1a: Structure of Generator G Input 256x256x3 image 32 convolutions 4x4. Stride 2. Leaky RELU, 64 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 128 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 convolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 256 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 128 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 64 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 32 deconvolutions 4x4, stride 2. Instance normalization Leaky RELU, 3 deconvolutions 4x4, stride 2. Tanh

[0065] Figure 3Outlines the architecture of discriminator D1 or D2. Layer 31 is the input layer of an embodiment of discriminator D1 or D2. Each discriminator may include multiple layer blocks. It may include a convolutional layer with N10 (e.g., N10 = 64) kernels and a stride of 2, leaky ReLU; a convolutional layer with N11 (e.g., N11 = 128) kernels and a stride of 2, leaky ReLU; a convolutional layer with N12 (e.g., N12 = 256) kernels and a stride of 2, leaky ReLU; a convolutional layer with N13 (e.g., N13 = 512) kernels and a stride of 2, leaky ReLU; and a convolutional layer or an optional softmax layer. Table 1b describes these layers in more detail.

[0066] Table 1b: Structure of Discriminator D1 / D2 Input 256x256x3 image 64 convolutions 4x4, stride 2, Leaky RELU 128 convolutions 4x4, stride 2, Instance normalization, Leaky RELU 256 convolutions 4x4, stride 2, Instance normalization, Leaky RELU 512 convolutions 4x4, stride 2, Instance normalization, Leaky RELU 1 convolution 4x4, stride 1.

[0067] Train an artificial neural generator network to generate an intermediate generator image, which can be superimposed on a pure image and transfer the features of a feature image to a receiving image to construct a pseudo-feature image including the feature. To do so, the generator needs to learn the appearance of the feature image, which means the generator needs to learn to transfer the feature image including the feature to an output intermediate generator image that only represents the feature. Superimpose this intermediate generator image on an image without the feature to generate a new pseudo-feature image including the feature.

[0068] Figure 4 Describes the steps of training an artificial neural generator network to transfer the feature of a feature image including the feature to a pure image.

[0069] Before starting training, set each parameter of the network to have an initial value. There are several initialization methods, such as setting the initial value of the parameter to 0, randomly selecting a value from a Gaussian distribution of N(0,1), or using a certain standard parameter initialization method (such as xavier initialization).

[0070] At step S1, if a feature image among multiple feature images including the feature and a pure image without the feature among multiple pure images without the feature are provided to the input of the generator network, the generator network generates an intermediate generator image.

[0071] At step S2 of the described method, the intermediate generator image and the feature image are superimposed to construct a pseudo-image.

[0072] At step S3, the pseudo-image and the pure image are provided to the first discriminator network.

[0073] At step S4, calculate the loss function of the first discriminator network. The loss function of D1 is defined as:

[0074]

[0075] In this equation, x2 indicates the image with the attribute, and x1 indicates the image without the attribute. This loss L1 consists of two parts. The first term is the loss of the real image without this feature. The second term is the loss of the pseudo-image generated by adding the generated residual image G(x1, x2) to x2.

[0076] In step S5, the inverted intermediate generator image is superimposed with the pure image to construct a pseudo-feature image.

[0077] In step S6, the pseudo-feature image and the feature image are provided to the second discriminator network.

[0078] In step S7, the loss function of the second discriminator network is calculated. The loss function of D2 is defined to be similar to the loss function of D1 explained above:

[0079]

[0080] In method step S8, if the number of training sequences counted for the implemented training sequence is odd, the network parameters of the first and second discriminator networks are recalculated.

[0081] In step S9, the loss function of the generator network is calculated. The loss function of generator G is defined as:

[0082]

[0083] This loss function consists of three parts. The first term is the loss defined for D1. The second term is the loss defined by D2. The third term is the L1 normalization of the generated residual map, which encourages the generated residual map to be sparse.

[0084] In step S10, if the number of training sequences is even, the network parameters of the generator network are recalculated. This means that during training, the parameters of the generator and discriminator are updated iteratively.

[0085] In step S11, the absolute value of the loss function of the generator network is compared with a predefined threshold to determine whether to continue with step 1 or end the training sequence.

[0086] Figure 5 A data stream for transferring the feature of a feature image including a feature to a receiving image is described. After training the generator network (G)1 as described above, the generator network (G)1 can generate a pseudo-image by transferring the feature of the feature image I+ to another image such as a receiving image.

[0087] A feature image I+ (i.e., an image including the features to be transferred) and a receiving image (i.e., an image that should be provided with the features) are provided at the input 1a or 1b of the generator network (G) 1.

[0088] The input data of the generator network (G) 1 can be constructed in the same manner as the training method of the generator network described above.

[0089] As Figure 5 indicated, the generator output 1c can provide an intermediate generator image as its input and the result of the trained generator network (G) 1, and can be superimposed with the feature image I+ to construct a generated pseudo-image without the feature based on the feature image.

[0090] After inverting the intermediate generator image as Figure 5 indicated, it is superimposed with the receiving image I to provide a generated pseudo-feature image including the feature and based on the receiving image.

[0091] As is clear, the structure of the generator network (G) 1 is the same as the structure of the generator network (G) used for training, and has been described in detail above.

Claims

1. A method for training an artificial neural generator network (1), which transfers the features of a feature image including features to a pure image by repeating a training sequence of the generator network (1) until the absolute value of the loss function of the generator network (1) is lower than a predefined threshold (S11). The training sequence includes: Generating an intermediate generator image (S1) by providing a feature image among a plurality of feature images including the features and a pure image without the features among a plurality of pure images without the features to the generator network; Superimposing the intermediate generator image with the feature image (S2) to construct a pseudo-image; Providing the pseudo-image and the pure image to a first discriminator network (2) (S3); Calculating the loss function of the first discriminator network (2) in any odd-numbered training sequence (S4); Superimposing the inverted intermediate generator image with the pure image (S5) to construct a pseudo-feature image; Providing the pseudo-feature image and the feature image to a second discriminator network (3) (S6); Calculating the loss function of the second discriminator network (3) in any odd-numbered training sequence (S7); Recalculating the network parameters of the first discriminator network (2) and the second discriminator network (3) in any odd-numbered training sequence (S8); Calculating the loss function of the generator network (1) in any even-numbered training sequence (S9); and Recalculating the network parameters of the generator network (1) in any even-numbered training sequence (S10).

2. The method according to claim 1, wherein, Calculating the loss function of the generator (S9) takes into account the output values of the first discriminator (2) and the second discriminator (3).

3. The method according to one of the preceding claims, wherein, Calculating the loss function of the generator (S9) using the following formula: where E x1~p(x) is the expectation of the true image distribution without attributes, and E x2~p(x+) is the expectation of the true image distribution with attributes, x2 indicates an image with attributes, and x1 indicates an image without attributes.

4. The method according to one of claims 1 to 2, wherein, The artificial neural generator network (1) is a convolutional encoder-decoder neural network.

5. A method for transferring the features of a feature image including features to a received image by means of the artificial neural generator network (1) trained according to one of claims 1 to 4, the method including: Providing the feature image and the received image to the generator network (S1), and the generator network (10) provides the feature image; Superimposing the inverted feature image with the received image (S5) to transfer the features of the feature image including the features to the received image.

6. A method for generating a pseudo-feature image including features, the method including: Iteratively implementing the method of transferring the features of the feature image according to claim 5, wherein A feature image including the features selected from a first plurality of feature images is repeatedly provided to the generator network (1), and each received image selected from a second plurality of received images is uniquely provided to the generator network (1), and wherein the number of elements in the second plurality is higher than the number of elements in the first plurality.

7. A computer program product including instructions that, when executed by a computer, cause the computer to implement the method according to one of claims 1 to 6.

8. A computer-readable storage medium including instructions that, when executed by a computer, cause the computer to implement the method according to one of claims 1 to 6.

Citation Information

Patent Citations

  • Generative adversarial network-based image style transfer method

    CN107464210A

  • GNN-based cultural relic image restoration method

    CN107862668A