A lymphoma image data generation method based on conditional constraint and feature matching of a generative adversarial network

Abstract

The application provides a lymphoma image data generation method based on conditional constraint and feature matching of a generative adversarial network, and is used for data enhancement of lymphoma images. The method first combines a label constraint, builds a full connection layer and four deconvolution layers for a generative model of WGAN-GP to generate images, then builds three convolution layers and a full connection layer for a discriminant model of WGAN-GP to identify whether the images are from the real or generated, and finally combines a feature matching method to establish a loss function, generates images with specified labels through the adversarial learning of the generative model and the discriminant model, so as to achieve the purpose of data enhancement. A large number of clinical cases are needed for the acquisition of lymphoma images, however, the method can solve the problem that workers are difficult to acquire specified type lymphoma image data in the prior art, further improves the work efficiency of workers, thereby reduces the working time of workers and improves the authenticity of the generated images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer application, and specifically relates to a lymphoma image data generation method based on conditional constraint and feature matching of a generative adversarial network. BACKGROUND

[0002] Image recognition is a very important technology today, and lymphoma image recognition is an important application of image recognition. Deep learning is an effective method for realizing image recognition. Deep learning is a method for representing data in machine learning. Observation values (such as an image) can be represented in various ways, such as a vector of each pixel intensity value or a series of edges, a specific shape region, and the like, and some specific representation method is more likely to learn a task (such as face recognition or facial expression recognition) from instances. The advantage of deep learning is to replace manual feature acquisition with unsupervised or semi-supervised feature learning and hierarchical feature extraction algorithms. However, deep learning requires a large number of labeled images for learning. However, the acquisition of lymphoma image data often requires a large number of clinical cases, which is difficult to collect.

[0003] Image enhancement technology can effectively alleviate the above problems. Compared with the traditional image cutting and rotating data enhancement method, it is more reliable to generate images that meet the requirements by acquiring image features through deep learning. The generative adversarial network is a deep learning network based on an adversarial mode. After being trained by existing data, it can be used to generate a large number of data conforming to the sample distribution, thereby solving the problem of insufficient image data. WGAN-GP is a kind of generative adversarial network, which uses EM distance instead of the original JS divergence in the generative adversarial network, so that WGAN-GP can generate more realistic image data. However, in deep learning, not only a large number of lymphoma image data are needed, but also lymphoma images with labeled information are needed. The acquisition of a large number of lymphoma images is not only difficult, but also requires staff to identify the images, which requires a lot of working time and greatly affects work efficiency. Therefore, generating lymphoma images of a specified category in an intelligent manner can greatly improve the work efficiency of staff.

[0004] Due to these challenges and the outstanding performance of WGAN-GP in image generation, a method for generating lymphoma image data using a generative adversarial network (GAN) was designed based on conditional constraints and feature matching. Conditional constraints enable WGAN-GP to generate lymphoma images of a specified label category. Feature matching calculates the feature difference between the generated image and the real image using a discriminant model. Convolutional layers are used to construct the generative and discriminative models, enabling them to better learn the features of real lymphoma images. By minimizing the optimization loss, the generated lymphoma images appear more realistic. Summary of the Invention

[0005] To address the problem that labeled lymphoma images are difficult to obtain, the present invention proposes a lymphoma image data generation method based on a generative adversarial network with conditional constraints and feature matching.

[0006] The technical solution of the present invention is: a method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching, comprising the following steps:

[0007] Step 1: Normalize the image data and perform OneHot encoding on the class label data;

[0008] Step 2: Build a WGAN-GP generative model based on a convolutional neural network according to the image class label information;

[0009] Step 3: Build a WGAN-GP discriminant model based on a convolutional neural network according to the image class label information;

[0010] Step 4: Calculate the penalty term for feature matching based on the values ​​of the real sample and the generated sample in each layer of the discriminant model;

[0011] Step 5: Construct loss functions for the generative model and the discriminative model;

[0012] Step 6: Optimize the loss and introduce image data to train the model;

[0013] Step 7: Generate the expected image by specifying the class label based on the trained generative model.

[0014] 9. Further, the step 1 comprises the following steps:

[0015] Step 1.1: Load the image and normalize it to limit the value to the interval [0,1].

[0016] Step 1.2: Use the OneHotEncoder method in the Python sklearn package to encode the class label corresponding to the image.

[0017] 10. Further, the step 2 comprises the following steps:

[0018] Step 2.1: Initialize the input noise. Import the tensorflow library into Python and use the normal distribution random value method to initialize the noise with a dimension of [batch_size, 128]. The first parameter batch_size is the number of samples that the generative model can generate per generation; the second parameter 128 means that every 128-dimensional noise will generate a sample through the generative model.

[0019] Step 2.2: Combine the noise and conditional constraints. Use the concat method in TensorFlow to concatenate the noise and labels. The dimension of the label is [batch_size, label_size]. The label_size is the value of the class label after OneHot encoding. The concatenated value is used as the input of the generative model, and its dimension is [batch_size, 128 + label_size].

[0020] Step 2.3: Build a fully connected neural network. Import the fully_connected method in the tensorflow.contrib.layers package to construct a fully connected layer. The input dimension is the concatenation of the noise and label dimensions [batch_size, 128 + label_size], and the output dimension is 16*16*8*DEPTH, where DEPTH is the length of the image. The weights of the fully connected network are initialized with values ​​that follow a normal distribution with a standard deviation of 0.01. The bias item is initialized to 0.0. Then, the values ​​of the fully connected network are reshaped using the tensorflow reshape method to change the dimension values ​​to [batch_size, 16, 16, 8*DEPTH]. The value of the first parameter batch_size is the same as in step 2.1. The second and third parameters of 16 indicate the size of the output image. The fourth parameter 8*DEPTH indicates the number of channels.

[0021] Step 2.4: Build four deconvolution layers. Use the conv2d_transpose method in tensorflow to build the deconvolution layer. The input dimension of the first deconvolution layer is the output of the fully connected layer. The dimension of the convolution kernel is [5,5,4*DEPTH,8*DEPTH], the convolution step dimension is [1,2,2,1], and the convolution kernel is initialized with a standard deviation of 0.01. The value of the normal distribution is obtained by the initialization of the bias term is set to 0.0, and the padding value is set to "SAME"; the convolution value is passed through the relu activation function, and then the output of the relu activation function is reshaped using the reshape method in tensorflow to change the dimension to [batch_size, 32, 32, 4*DEPTH]; the input of the second deconvolution layer is the output after the reshape method, where the dimension of the convolution kernel is [5, 5, 2*DEPTH, 4*DEPTH], the convolution step dimension is [1, 2, 2, 1], the convolution kernel is initialized with a value that obeys the normal distribution with a standard deviation of 0.01, the bias initialization value is set to 0.0, and the padding value is set to "SAME"; the convolution value is passed through the relu activation function, and the first The input of the three deconvolution layers is the output of the relu activation function, the dimension of the convolution kernel is [5,5,DEPTH,2*DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized to a value that obeys a normal distribution with a standard deviation of 0.01, the bias item is initialized to 0.0, and the padding value is set to "SAME". The convolution value is passed through the relu activation function. The input of the fourth deconvolution layer is the output of the relu activation function, the dimension of the convolution kernel is [5,5,1,DEPTH], the convolution step dimension is [1,2,2,1]. The convolution kernel is initialized to a value that obeys a normal distribution with a standard deviation of 0.01, the bias item is initialized to 0.0, and the padding value is set to "SAME". The convolution value is passed through the sigmoid activation function.

[0022] Step 2.5: Output the generated samples. Use the reshape method in the tensorflow library to change the output dimension of the sigmoid activation function to [-1, img_size]. The first parameter -1 indicates that the samples are automatically arranged into a row according to the second parameter. img_size indicates the size of the image. The converted value is used as the output of the generative model.

[0023] 11. Further, the step 3 comprises the following steps:

[0024] Step 3.1: Connection label constraint, reshape the input image information dimension to [-1, img_length, img_width, 1] in tensorflow, where the first parameter -1 represents data according to the last three parameters, the second parameter and the third parameter represent the length and width of the image respectively, and the fourth parameter 1 represents the channel number, and the label dimension is also reshaped to [-1, 1, 1, label_size] by using the reshape method, label_size represents the label dimension after OneHot encoding, then the label and image are connected by using the concat method in tensorflow, and the dimension becomes [batch_size, img_length, img_width, label_size+1];

[0025] Step 3.2: Build three layers of convolutional layers, use the conv2d method in tensorflow to build convolutional layers, the input dimension of the first layer of convolutional layer is the dimension after the label and image are connected, the convolution kernel dimension is [5, 5, label_size+1, DEPTH], the convolution step dimension is [1, 2, 2, 1], the convolution kernel is initialized with values subject to normal distribution with standard deviation 0.01, the bias term initialization value is set to 0.01, the padding value is set to "SAME", and the value after convolution is activated by the leaky_relu activation function. The input of the second layer is the output of the activation function of the last layer, the convolution kernel dimension is [5, 5, DEPTH, 2*DEPTH], the convolution step dimension is [1, 2, 2, 1], the convolution kernel is initialized with values subject to normal distribution with standard deviation 0.01, the bias term initialization value is set to 0.01, the padding value is set to "SAME", and the value after convolution is activated by the leaky_relu activation function. The input of the third layer is the output of the activation function of the last layer, the convolution kernel dimension is [5, 5, 2*DEPTH, 4*DEPTH], the convolution step dimension is [1, 2, 2, 1], the convolution kernel is initialized with values subject to normal distribution with standard deviation 0.01, the bias term initialization value is set to 0.01, the padding value is set to "SAME", and the value after convolution is activated by the leaky_relu activation function, and a d_list list is created to store the values after convolution of each layer of convolutional network, so as to calculate the feature matching penalty term later;

[0026] Step 3.4: Construct a fully connected layer. First, use the reshape function in tensorflow to transform the dimension of the convolutional layer output value into [batch_size, channel[1]*channel[2]*channel[3]], where the value of the first parameter batch_size is the same as that in step 2.1, and the second parameter channel[1], channel[2], and channel[3] represent the second, third, and fourth parameters of the output layer dimension of the previous network layer, respectively. After the dimension is transformed in the above way, the output dimension can be made 1 by the fully connected layer, making it easier for the model to judge the quality of the generated samples.

[0027] Step 3.5: The discriminator outputs the fully connected output and the list of values ​​after storage convolution.

[0028] 12. Further, the step 4 comprises the following steps:

[0029] Calculate the feature matching penalty. The list returned by the discriminant model can be used to know the values ​​of the real samples and the generated samples passing through the discriminant model convolution layer. Calculate the square sum of the values ​​output by the corresponding convolution layers of the two lists and use the reduce_mean method in tensorflow to calculate the mean. Then, the mean calculated for all elements of the list is accumulated to obtain the feature matching penalty feature_loss.

[0030] 13. Further, the step 5 comprises the following steps:

[0031] Step 5.1: Calculate the discriminant model's discriminant values ​​for the generated and real samples. Load the training samples, place the training sample labels into the generative model, and generate samples using the generative model. Place the generated samples and labels into the discriminant model to obtain the discriminant value disc_fake for the generated samples. Place the real samples and labels into the discriminant model to obtain the discriminant value disc_real for the real samples.

[0032] Step 5.2: Calculate the loss of the generative model. The loss of the generative model consists of two parts: the judgment value disc_fake of the generated sample by the discriminant model, the penalty value obtained by the reduce_mean method in tensorflow, and the feature matching penalty term feature_loss obtained in step 4. Add these two items to get the loss of the generative model;

[0033] Step 5.3: Calculate the gradient penalty term of the discriminant model. The gradient penalty term formula is as follows:

[0034]

[0035] Where λ represents the gradient penalty coefficient and its value is 10. Represents the mean of the selected samples, and x represents the distribution For a randomly selected sample, the p-value is 2. Indicates that the discriminant model D(x) is derived from x, The value is obtained according to the following formula:

[0036]

[0037] ε represents a random coefficient between 0 and 1, x r represents a sample in the true sample distribution, x g Represents a sample from the generated sample distribution;

[0038] The gradient penalty term gradient_penalty can be obtained through the above formula;

[0039] Step 5.4: Calculate the loss of the discriminant model. The loss of the discriminant model consists of three parts. The first part is the discriminant model's generated sample's judgment value disc_fake, and the penalty value (here with a negative sign) calculated by the reduce_mean method in TensorFlow. The second part is the discriminant model's real sample's judgment value disc_real, and the penalty value calculated by the reduce_mean method in TensorFlow. The third part is the gradient penalty term calculated in step 5.3. Add these three parts together to get the discriminant model loss.

[0040] 14. Further, the step 6 comprises the following steps:

[0041] Step 6.1: Optimize the loss. Use RMSProp to optimize the loss of the generative model and the discriminative model. The learning rate learning_rate is set to 0.0001, and the minimization optimization method is adopted.

[0042] Step 6.2: Load the data training model, select 50 training iterations epoch, use batch processing to train, select batch_size samples in each batch, train 1000 times per iteration, train the discriminant model five times before each training of the generative model, and terminate the training when the number of training iterations reaches 50.

[0043] 15. Further, the step 7 comprises the following steps:

[0044] By obtaining a trained generative model and passing specified labels, the corresponding lymphoma images can be generated, thereby achieving the purpose of data enhancement for lymphoma image data.

[0045] Finally, in order to verify the feasibility of the method, the lymphoma image data generation method of the generative adversarial network based on the constraint condition and the feature matching is tested on the three-class lymphoma data set.

[0046] The present application has the following technical effects:

[0047] Step 2, the present application combines random noise and label constraints to form the input of the generation model, adopts a fully connected layer and four deconvolution layers to construct the generation model, so that the generation model can better capture the detailed information of the real image, thereby being able to generate more real images. By inputting the specified label, the generation model can generate images that meet the requirements, thereby achieving the purpose of data enhancement.

[0048] Step 3, the present application combines the input image and the label constraint to form the input of the discriminant model, adopts three convolution layers and a fully connected layer to build the discriminant model, so that the discriminant model can better distinguish whether the input image is real or generated, thereby being able to force the generation model to learn more complete real information and improve the image sample quality of the generation model.

[0049] Step 4, the present application adopts the feature matching idea, calculates the feature value difference by recording the feature values of the generated samples and the real samples through the convolution layer of the discriminant model, forms the feature loss, and reflects the difference between the generated samples and the real samples. The generation model can better learn the features of the real samples through the feature loss, thereby generating more real images.

[0050] As can be seen, the method can generate image data that meets the requirements and has good quality by inputting the specified label, thereby being able to improve the efficiency of the staff in obtaining the specified label image data. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of a lymphoma image data generation method of a generative adversarial network based on conditional constraints and feature matching in the present application.

[0052] Figure 2 (a) is a real image of a lymphoma data set.

[0053] Figure 2 (b) is a lymphoma image generated by the algorithm. DETAILED DESCRIPTION

[0054] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present application. It should be particularly noted that in the following description, detailed descriptions of known functions and designs may dilute the main content of the present application, and these descriptions will be omitted here.

[0055] like Figure 1 As shown, the present invention provides a method for generating lymphoma image data using a generative adversarial network based on conditional constraints and feature matching, comprising the following steps:

[0056] Step 1: Normalize the image data and perform OneHot encoding on the class label data.

[0057] Step 2: Build a WGAN-GP generation model based on convolutional neural network according to the image class label information.

[0058] Step 3: Build a WGAN-GP discriminant model based on convolutional neural network according to the image class label information.

[0059] Step 4: Calculate the penalty term for feature matching based on the values ​​of the real samples and the generated samples in each layer of the discriminant model.

[0060] Step 5: Construct loss functions for the generative model and the discriminative model.

[0061] Step 6: Optimize the loss and introduce image data to train the model.

[0062] Step 7: Generate the expected image by specifying the class label based on the trained generative model.

[0063] The flowchart of the whole process is as follows Figure 1 shown.

[0064] The lymphoma image data generation method of the present invention based on a generative adversarial network with conditional constraints and feature matching, wherein step 1 comprises the following steps:

[0065] Step 1.1: Load the image and normalize it to the interval [0, 1]. Use the MinMaxScaler method in the sklearn package to normalize the image data to the interval [0, 1], which will help the model learn better.

[0066] Step 1.2: Use the OneHotEncoder method from the Python sklearn package to encode the class labels corresponding to the images. OneHot encoding represents each integer value of the class label as a binary vector. OneHot encoding can expand the values ​​of discrete features into Euclidean space, making distance calculations between features more reasonable and leading to more accurate training models.

[0067] Step 2 above includes the following steps:

[0068] Step 2.1: Initialize the input noise. Import the TensorFlow library into Python and initialize the noise with dimensions [batch_size, 128] using a normal distribution. The first parameter, batch_size, is the number of samples the generative model can generate per generation. Here, a value of 100 indicates that the generative model can generate 100 samples at a time. The second parameter, 128, indicates that the generative model will generate one sample for every 128-dimensional noise.

[0069] Step 2.2: Combine the noise and the constraints. Use the concat method in TensorFlow to concatenate the noise and labels, where the labels have the dimension [batch_size, label_size]. label_size is the value of the class label after OneHot encoding. This concatenation serves as the input to the generative model, with the dimension [batch_size, 128 + label_size]. The batch_size is set to 100, and label_size is the number of image categories.

[0070] Step 2.3: Build a fully connected neural network. Import the fully_connected method from the tensorflow.contrib.layers package to construct a fully connected layer. The input dimension is the concatenation of the noise and label dimensions [batch_size, 128 + label_size], and the output dimension is 16*16*8*DEPTH, where DEPTH is the image length. For example, if an image is 256*256*3, DEPTH is 256. The weights of the fully connected network are initialized with values ​​from a normal distribution with a standard deviation of 0.01, and the bias is initialized to 0.0. Then, use the reshape method in TensorFlow to reshape the values ​​of the fully connected network to [batch_size, 16, 16, 8*DEPTH]. The first parameter, batch_size, has the same value as in step 2.1. The second and third parameters, 16, indicate the output image size. The fourth parameter, 8*DEPTH, indicates the number of channels.

[0071] Step 2.4: Build four deconvolution layers. Use the conv2d_transpose method in TensorFlow to build the deconvolution layers. The first deconvolution layer uses the output of the fully connected layer as its input. The kernel dimensions are [5, 5, 4*DEPTH, 8*DEPTH], and the stride dimensions are [1, 2, 2, 1]. The kernel is initialized to a normal distribution with a standard deviation of 0.01. The bias is initialized to 0.0, and the padding value is set to "SAME." The convolved value is passed through the ReLU activation function. The output of the ReLU activation function is then reshaped using the reshape method in TensorFlow to change the dimensions to [batch_size, 32, 32, 4*DEPTH]. The second deconvolution layer's input is the output of the reshape method. The convolution kernel dimensions are [5, 5, 2*DEPTH, 4*DEPTH], the convolution stride dimensions are [1, 2, 2, 1], the convolution kernel is initialized to values ​​from a normal distribution with a standard deviation of 0.01, the bias is initialized to 0.0, and the padding value is set to "SAME". The convolution value is then passed through the ReLU activation function. The third deconvolution layer's input is the output of the ReLU activation function. The convolution kernel dimensions are [5, 5, DEPTH, 2*DEPTH], the convolution stride dimensions are [1, 2, 2, 1], the convolution kernel is initialized to values ​​from a normal distribution with a standard deviation of 0.01, the bias is initialized to 0.0, and the padding value is set to "SAME". The convolution value is then passed through the ReLU activation function. The input to the fourth deconvolution layer is the output of the ReLU activation function. The convolution kernel dimensions are [5, 5, 1, DEPTH], the convolution stride dimensions are [1, 2, 2, 1], and the kernel is initialized to a normal distribution with a standard deviation of 0.01. The bias is initialized to 0.0, and the padding value is set to "SAME". The convolved value is then passed through the sigmoid activation function.

[0072] Step 2.5: Output the generated samples. Use the reshape method in the TensorFlow library to reshape the output dimension of the sigmoid activation function to [-1, img_size]. The first parameter, -1, indicates that the images are automatically arranged into a row according to the second parameter. img_size represents the image size. For example, if the image size is 256*256*3, then img_size is 196608. Use this converted value as the output of the generative model.

[0073] Step 3 above includes the following steps:

[0074] Step 3.1: Connect the label constraints. Use the reshape method in TensorFlow to change the dimensions of the input image information to [-1, img_length, img_width, 1], where the first parameter -1 indicates that the data is divided according to the last three parameters, the second and third parameters represent the length and width of the image, respectively, and the fourth parameter 1 represents the number of channels. Similarly, use the reshape method to change the dimensions of the label to [-1, 1, 1, label_size], where label_size represents the label dimension after OneHot encoding. Then, use the concat method in TensorFlow to concatenate the label and image, and the dimensions become [batch_size, img_length, img_width, label_size+1]. For example, if the input image size is 256*256*3 and the image is a 3-category image, then the values ​​of img_length and img_width are 256, the number of channels is 3, and the processed dimensions become [100, 256, 256, 4].

[0075] Step 3.2: Construct three convolutional layers. Use the conv2d method in TensorFlow to construct the convolutional layers. The input dimensions of the first convolutional layer are the concatenated dimensions of the label and image. The convolution kernel dimensions are [5, 5, label_size + 1, depth], and the convolution stride dimensions are [1, 2, 2, 1]. The convolution kernel is initialized to a value that follows a normal distribution with a standard deviation of 0.01. The bias is initialized to 0.01, and the padding value is set to "same". The convolutional value is activated by the leaky_relu function. The input of the second layer is the output of the activation function of the previous layer. The convolution kernel dimensions are [5, 5, depth, 2* depth], and the convolution stride dimensions are [1, 2, 2, 1]. The convolution kernel is initialized to a value that follows a normal distribution with a standard deviation of 0.01. The bias is initialized to 0.01, and the padding value is set to "same". The convolutional value is activated by the leaky_relu function. The input to the third layer is the output of the activation function in the previous layer. The convolution kernel dimensions are [5, 5, 2*DEPTH, 4*DEPTH], the convolution stride dimensions are [1, 2, 2, 1], and the kernel is initialized to a normally distributed value with a standard deviation of 0.01. The bias is initialized to 0.01, and the padding value is set to "SAME". The convolved value is passed through the leaky_relu activation function. A list d_list is created to store the convolutional values ​​of each layer of the convolutional network, which is then used to calculate the feature matching penalty.

[0076] Step 3.4: Construct a fully connected layer. First, use the reshape function in TensorFlow to transform the dimensions of the convolutional layer output value to [batch_size, channel[1]*channel[2]*channel[3]], where the first parameter is the same as in step 2.1, and the second parameter is the product of the second, third, and fourth parameters of the output layer of the previous network. After the dimension transformation, the output passes through the fully connected layer, and the dimension is 1.

[0077] Step 3.5: Discriminator output. Output the fully connected output and the list d_list storing the values ​​after convolution to facilitate the calculation of feature loss later.

[0078] Step 4 above includes the following steps:

[0079] Calculate the feature matching penalty. The list returned by the discriminant model shows the values ​​of the real and generated samples passing through the discriminant model's convolutional layer. Calculate the square sum of the corresponding convolutional layer output values ​​in the two lists and use the reduce_mean method in TensorFlow to find the mean. Then, sum the calculated means of all elements in the lists to obtain the feature matching penalty term, feature_loss. The relevant formula is as follows:

[0080]

[0081] Represents the value of the generated sample after passing through the i-th convolutional layer of the discriminator, Represents the value of the real sample after passing through the i-th convolutional layer of the discriminator.

[0082] Step 5 above includes the following steps:

[0083] Step 5.1: Calculate the discriminant model's discriminant values ​​for the generated and real samples. Load the training samples, insert their labels into the generative model, and generate samples using the generative model. Insert the generated samples and labels into the discriminant model to obtain the discriminant value disc_fake for the generated samples. Insert the real samples and labels into the discriminant model to obtain the discriminant value disc_real for the real samples.

[0084] Step 5.2: Calculate the loss of the generative model. The loss of the generative model consists of two parts. One is the discriminant model's generated sample judgment value disc_fake, the penalty value obtained by the reduce_mean method in TensorFlow, and the feature matching penalty term feature_loss obtained in step 4. Adding these two terms gives the loss of the generative model. The loss function formula of the generative model is as follows:

[0085] g_loss=-E(disc_fake)+feature_loss

[0086] g_loss is the loss function of the generative model.

[0087] Step 5.3: Calculate the gradient penalty term of the discriminant model. The gradient penalty term formula is as follows:

[0088]

[0089] Where λ represents the gradient penalty coefficient, which is 10, and x represents the gradient penalty coefficient. A sample randomly selected from . The p-value is 2. represents the derivative of x. The value is obtained according to the following formula:

[0090]

[0091] ε represents a random coefficient between 0 and 1. r represents a sample from the true sample distribution. g represents a sample from the generative sample distribution.

[0092] The gradient penalty term gradient_penalty can be calculated through the above formula.

[0093] Step 5.4: Calculate the loss of the discriminant model. The loss of the discriminant model consists of three parts. The first part of the loss is the penalty value obtained by the reduce_mean method in tensorflow through the judgment value disc_fake of the generated sample of the discriminant model (a negative sign is added here). The second part of the loss is the penalty value obtained by the reduce_mean method in tensorflow through the judgment value disc_real of the real sample of the discriminant model. The third part of the loss is the gradient penalty term obtained in step 5.3. Add these three parts to get the discriminant model loss. The formula of the discriminant model loss function is as follows:

[0094] d_loss=E(disc_fake)-E(disc_real)+gradient_penalty

[0095] d_loss is the loss function of the discriminant model.

[0096] Step 6 above includes the following steps:

[0097] Step 6.1: Optimize the loss. Use the RMSProp method in TensorFlow to optimize the loss of the generative and discriminative models. Set the learning rate learning_rate to 0.0001 and use a minimization optimization method.

[0098] Step 6.2: Load the data and train the model. Set the number of training epochs to 50. Use batch training, selecting batch_size samples per batch. Train 1000 times per iteration. Train the discriminant model five times before each generative model training. Terminate training after 50 training iterations.

[0099] Step 7 above includes the following steps:

[0100] Obtain a trained generative model and generate corresponding images by passing specified labels, thereby achieving the purpose of data enhancement for image data.

[0101] Finally, in order to verify the feasibility of the algorithm, the present invention tests the lymphoma image data generation method based on the generative adversarial network with conditional constraints and feature matching on a three-class lymphoma dataset. The test results are shown in the figure of the specification. Figure 2 (b) shown.

[0102] In summary, the present invention provides a method for generating lymphoma image data using a generative adversarial network based on conditional constraints and feature matching, which is used to obtain lymphoma images with specified labels. This method first combines random noise with label constraints to form the input of the generative model. A generative model is constructed using one fully connected layer and four deconvolution layers to capture the distribution of real image data. The input image and label constraints are then combined to form the input of the discriminative model. The discriminative model is constructed using three convolutional layers and one fully connected layer, forcing the generative model to learn more complete real image information and improving the quality of the generative model's image samples. Finally, a feature matching approach is employed. By recording the eigenvalues ​​of the generated and real samples as they pass through the convolutional layer of the discriminative model, the eigenvalue differences are calculated to form a feature loss. This feature loss allows the generative model to better learn the features of the real samples, thereby generating more realistic images.

Claims

1. A method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching, characterized in that: The following steps are involved: Step 1: Normalize the image data and perform OneHot encoding on the class label data; Step 2: Build a WGAN-GP generative model based on a convolutional neural network according to the image class label information; Step 3: Build a WGAN-GP discriminant model based on a convolutional neural network according to the image class label information; Step 4: Calculate the penalty term for feature matching based on the values ​​of the real sample and the generated sample in each layer of the discriminant model; Step 5: Construct loss functions for the generative model and the discriminative model; Step 6: Optimize the loss and introduce image data to train the model; Step 7: Generate the expected image by specifying the class label based on the trained generative model; The step 1 comprises the following steps: Step 1.1: Load the image and normalize it to limit the value to the interval [0,1]. Step 1.2: Use the OneHotEncoder method in the Python sklearn package to encode the class label corresponding to the image; The step 2 comprises the following steps: Step 2.1: Initialize the input noise. Import the tensorflow library into Python and use the normal distribution random value method to initialize the noise with a dimension of [batch_size, 128]. The first parameter batch_size is the number of samples that the generative model can generate per generation; the second parameter 128 means that every 128-dimensional noise will generate a sample through the generative model. Step 2.2: Combine the noise and conditional constraints. Use the concat method in TensorFlow to concatenate the noise and labels. The dimension of the label is [batch_size, label_size]. The label_size is the value of the class label after OneHot encoding. The concatenated value is used as the input of the generative model, and its dimension is [batch_size, 128 + label_size]. Step 2.3: Build a fully connected neural network. Import the fully_connected method in the tensorflow.contrib.layers package to construct a fully connected layer. The input dimension is the concatenation of the noise and label dimensions [batch_size, 128 + label_size], and the output dimension is 16*16*8*DEPTH, where DEPTH is the length of the image. The weights of the fully connected network are initialized with values ​​that obey a normal distribution with a standard deviation of 0.01, and the bias item is initialized to 0.

0. Then, the values ​​of the fully connected network are reshaped using the tensorflow reshape method to change the dimension values ​​to [batch_size, 16, 16, 8*DEPTH]. The value of the first parameter batch_size is the same as in step 2.

1. The second and third parameters of 16 indicate the size of the output image, and the fourth parameter 8*DEPTH indicates the number of channels. Step 2.4: Build four deconvolution layers and use the conv2d_transpose method in tensorflow to build the deconvolution layer. The input dimension of the first deconvolution layer is the output of the fully connected layer, the dimension of the convolution kernel is [5,5,4*DEPTH,8*DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized using a normal distribution with a standard deviation of 0.01, the bias item is initialized to 0.0, and the padding value is set to "SAME"; the convolution value is passed through the relu activation function, and then the output of the relu activation function is reshaped using the reshape method in tensorflow to change the dimension to [batch_size,32,32,4*DEPTH]; the input of the second deconvolution layer is the output after the reshape method, where the dimension of the convolution kernel is [5,5,2*DEPTH,4*DEPTH], the convolution step dimension is [1,2,2,1], and the convolution kernel is initialized using Use the value of the normal distribution with a standard deviation of 0.01, the bias initialization value is set to 0.0, and the padding value is set to "SAME"; pass the convolution value through the relu activation function, the input of the third deconvolution layer is the output of the relu activation function, the dimension of the convolution kernel is [5,5,DEPTH,2*DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized with the value of the normal distribution with a standard deviation of 0.01, the bias initialization value is set to 0.0, and the padding value is set to "SAME"; pass the convolution value through the relu activation function, the input of the fourth deconvolution layer is the output of the relu activation function, the dimension of the convolution kernel is [5,5,1,DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized with the value of the normal distribution with a standard deviation of 0.01, the bias initialization value is set to 0.0, and the padding value is set to "SAME", and pass the convolution value through the sigmoid activation function; Step 2.5: Output the generated samples. Use the reshape method in the tensorflow library to change the output dimension of the sigmoid activation function to [-1, img_size]. The first parameter -1 indicates that the samples are automatically arranged into a row according to the second parameter. img_size indicates the size of the image. The converted value is used as the output of the generative model.

2. The method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Connect the label constraints and use the reshape method in tensorflow to change the dimension of the input image information to [-1, img_length, img_width, 1], where the first parameter -1 indicates that the data is divided according to the last three parameters, the second and third parameters indicate the length and width of the image respectively, and the fourth parameter 1 indicates the number of channels. The reshape method is also used to change the dimension of the label to [-1, 1, 1, label_size], where label_size indicates the label dimension after OneHot encoding. Then, the concat method in tensorflow is used to connect the label and image, and the dimension becomes [batch_size, img_length, img_width, label_size+1]; Step 3.2: Construct three convolutional layers. Use the conv2d method in tensorflow to build the convolutional layers. The input dimension of the first convolutional layer is the dimension after the label and image are concatenated. The dimension of the convolution kernel is [5,5,label_size+1,DEPTH], the convolution step dimension is [1,2,2,1], and the convolution kernel is initialized with a value that obeys the normal distribution with a standard deviation of 0.

01. The initialization value of the bias term is set to 0.01, padding value is set to "SAME", the convolved value is passed through the leaky_relu activation function, the input of the second layer is the output of the activation function of the previous layer, the dimension of the convolution kernel is [5,5,DEPTH,2*DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized to a value that obeys the normal distribution with a standard deviation of 0.01, and the bias term initialization value is set to 0.01, padding value is set to "SAME", the convolved value is passed through the leaky_relu activation function, the input of the third layer is the output of the activation function of the previous layer, the dimensions of the convolution kernel are [5,5,2*DEPTH,4*DEPTH], the convolution step dimension is [1,2,2,1], the convolution kernel is initialized using a value that obeys a normal distribution with a standard deviation of 0.01, and the bias term initialization value is set to 0.01, padding value is set to "SAME", the convolved value is passed through the leaky_relu activation function, and the d_list list is created to store the convolution value of each layer of the convolutional network, so as to calculate the feature matching penalty term later; Step 3.4: Construct a fully connected layer. First, use the reshape function in tensorflow to transform the dimension of the convolutional layer output value into [batch_size, channel[1]*channel[2]*channel[3]], where the value of the first parameter batch_size is the same as that in step 2.1, and the second parameter channel[1], channel[2], and channel[3] represent the second, third, and fourth parameters of the output layer dimension of the previous network layer, respectively. After the dimension transformation, the output dimension can be made 1 by the fully connected layer, making it easier for the model to judge the quality of the generated samples. Step 3.5: The discriminator outputs the fully connected output and the list of values ​​after storage convolution.

3. The method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching according to claim 1, characterized in that: The step 4 comprises the following steps: Calculate the feature matching penalty. The list returned by the discriminant model can be used to know the values ​​of the real samples and the generated samples passing through the discriminant model convolution layer. Calculate the square sum of the values ​​output by the corresponding convolution layers of the two lists and use the reduce_mean method in tensorflow to calculate the mean. Then, the mean calculated for all elements of the list is accumulated to obtain the feature matching penalty feature_loss.

4. The method for generating lymphoma image data using a generative adversarial network based on conditional constraints and feature matching according to claim 1, characterized in that: The step 5 comprises the following steps: Step 5.1: Calculate the discriminant model's discriminant values ​​for the generated and real samples. Load the training samples, place the training sample labels into the generative model, and generate samples using the generative model. Place the generated samples and labels into the discriminant model to obtain the discriminant value disc_fake for the generated samples. Place the real samples and labels into the discriminant model to obtain the discriminant value disc_real for the real samples. Step 5.2: Calculate the loss of the generative model. The loss of the generative model consists of two parts: the judgment value disc_fake of the generated sample by the discriminant model, the penalty value obtained by the reduce_mean method in tensorflow, and the feature matching penalty term feature_loss obtained in step 4. Add these two items to get the loss of the generative model; Step 5.3: Calculate the gradient penalty term of the discriminant model. The gradient penalty term formula is as follows: Where λ represents the gradient penalty coefficient and its value is 10. Represents the mean of the selected samples, and x represents the distribution For a randomly selected sample, the p-value is 2. Indicates that the discriminant model D(x) is derived from x, The value is obtained according to the following formula: ε represents a random coefficient between 0 and 1, x r represents a sample in the true sample distribution, x g Represents a sample from the generated sample distribution; The gradient penalty term gradient_penalty can be obtained through the above formula; Step 5.4: Calculate the loss of the discriminant model. The loss of the discriminant model consists of three parts. The first part of the loss is the judgment value disc_fake of the generated sample through the discriminant model, and the penalty value obtained by the reduce_mean method in tensorflow (add a negative sign here). The second part of the loss is the judgment value disc_real of the real sample through the discriminant model, and the penalty value obtained by the reduce_mean method in tensorflow. The third part of the loss is the gradient penalty term obtained in step 5.

3. Add these three parts to get the discriminant model loss.

5. The method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching according to claim 1, characterized in that: The step 6 comprises the following steps: Step 6.1: Optimize the loss. Use RMSProp to optimize the loss of the generative model and the discriminative model. The learning rate learning_rate is set to 0.0001, and the minimization optimization method is adopted. Step 6.2: Load the data training model, select 50 training iterations epoch, use batch processing to train, select batch_size samples in each batch, train 1000 times per iteration, train the discriminant model five times before each training of the generative model, and terminate the training when the number of training iterations reaches 50.

6. The method for generating lymphoma image data based on a generative adversarial network with conditional constraints and feature matching according to claim 1, characterized in that: The step 7 comprises the following steps: By obtaining a trained generative model and passing specified labels, the corresponding lymphoma images can be generated, thereby achieving the purpose of data enhancement for lymphoma image data.

Images