A grayscale image colorization method based on generative adversarial network

Through the method based on the generative adversarial network, the COCO data set and U-net architecture are used to solve the problem of time-consuming and flexibility of colorization of grayscale images, and the rapid and fine colorization of grayscale images is achieved, reducing manual intervention. Generate high-quality color images.

CN114581552BActive Publication Date: 2024-11-15NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210252286.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-15
Publication Date
2024-11-15
Estimated Expiration
2042-03-15

Smart Images

  • Figure CN114581552B_ABST
    Figure CN114581552B_ABST
Patent Text Reader

Abstract

The present invention discloses a grayscale image colorization method based on a generative adversarial network. The method first selects a quantitative color picture group in the COCO image data set, decolorizes it and makes it into a training set. Secondly, a generative adversarial network architecture is constructed so that the generator model completes pre-training therein, and then the discriminant model and the pre-trained generative model are alternately trained, and the parameters are adjusted to obtain a trained model. After the test data is input into the model, the grayscale image colorization can be realized. The present invention has a great improvement in the optimization of the training method and the data set through the pre-training method and process of the generator, and can greatly shorten the training time on the basis of ensuring the generalization quality of the training quality and the final generated image, and is flexible; and the idea of ​​U-Net is used to train and test on the COCO data set, which can greatly reduce the disadvantages of the traditional method that manual intervention is required and it is difficult to perform fine coloring work at the pixel level of large-size images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image generation, and in particular to a grayscale image colorization method based on a generative adversarial network. Background Art

[0002] Grayscale image colorization algorithm is a hot topic in the current field of digital image processing and computer vision. It has broad application prospects in black and white film and television materials, animation sketches, ancient painting restoration, as well as medicine and aviation. Traditional colorization methods are mainly divided into two categories: local color diffusion method and reference image-based color transfer method. The local color diffusion method relies on artificially giving some calibrated color pixels to color the global image through diffusion or variation methods. However, this method requires artificial provision of initial color diffusion pixels, which brings inconvenience to the colorization process; the color transfer method is based on one or more reference images with similar scenes and similar color distributions, so that the color is transferred to the candidate image, but the artificially given reference images required by such methods are often difficult to find in complex scenes. As a method used in the field of image colorization, the traditional methods in the past are difficult to achieve good results and require manual intervention. Even so, it is difficult to perform fine coloring work at the pixel level of large-size images.

[0003] Recently, due to the rapid development and widespread application of deep learning, many studies have chosen to combine deep learning and image colorization. The grayscale image colorization process based on deep learning is divided into two stages: training stage and colorization stage. First, a large number of different types of color images are put into the model for training. After continuous weight updates, the loss function is minimized. After the model reaches the optimal value, the image to be colored can be input into the colorization model to obtain a colored image, which greatly promotes the development of the field of image colorization.

[0004] Generative Adversarial Nets (GANs) is an unsupervised learning training method that consists of two parts: a generator network D and a discriminator network G. The generator network is used to generate realistic samples, and the discriminator is used to distinguish between generated samples and original samples. The learning process of GANs is to train the discriminator D and the generator G at the same time. The goal of the generator G is to learn the distribution Pa on the data x. G samples the input variable z from the uniform distribution or Gaussian distribution Pz(z), and then maps the input variable z to the data space through another network; D is the discriminator, and its purpose is to identify whether the image comes from the training data or from G. Through iterative alternating training, the discriminant model completes the task of distinguishing whether the input sample comes from the real data or the generated model. At the same time, the generated model is trained to generate data that the discriminant model cannot distinguish. During training, the two models iteratively train and compete, and finally generate data that is closest to the data distribution learned by the two-part model.

[0005] In terms of training sets, the performance of the colorization network is affected by the size and type of the training set. A small-scale and single-type training set can more easily learn the color information of the sample, but the learned color information is limited and can only color images of similar types to the training set, with poor generalization ability. If you want to obtain a more general grayscale image colorization model, you need a large number of images of different categories as training sets. Although large-scale datasets require a longer training time, they can learn richer color information, and the model has strong generalization ability and can color various types of grayscale images.

[0006] At present, in a research paper on grayscale image colorization algorithm based on deep learning from Xidian University, the multimodal nature of color prediction is modeled by quantizing the ab color channels in the Lab color space. Since the values ​​of ab in natural images tend to be smaller, the classification rebalancing technique is used during training to balance those color categories with less frequent occurrences, so as to maintain the diversity of colors. This article uses a subset of the ImageNet dataset, mainly including various animals, cars, fruits, and outdoor scenes such as buildings and beaches, including a total of 310,000 pictures. Among them, 300,000 pictures are training sets, 5,000 pictures are validation sets, and 5,000 pictures are test sets. Its characteristics are that a large-scale dataset is used, and the number of both datasets and training sets is very large. Although the generalization of images is guaranteed, it also causes the training to take a long time and has poor flexibility. Summary of the invention

[0007] The purpose of the present invention is to provide a grayscale image colorization method based on a generative adversarial network to solve the problems of long time consumption and poor flexibility in the current technology.

[0008] The technical solution adopted by the present invention to solve the above technical problems is as follows:

[0009] A grayscale image colorization method based on a generative adversarial network is implemented in the following steps:

[0010] A. Select a quantitative color image group from the COCO image dataset, perform decolorization, and make it into a training set;

[0011] B. Construct a generative adversarial network architecture, including a generator model and a discriminator model. The generator model is formed by pre-training the Resnet18 backbone network;

[0012] C. Input the training set obtained in step A into the generative adversarial network architecture in turn for model training and adjust the parameters to achieve convergence;

[0013] D. Preprocess the image to be processed and input the adversarial model obtained in step C to automatically colorize the grayscale image.

[0014] Preferably, in step A, a subset of the COCO dataset is selected, from which 8,000 images are selected as an optimized training set, 2,000 images are selected as a validation set, and 2,000 images are selected as a test set.

[0015] Preferably, the decolorization method of the training set in step A is to convert the color mode of the image from RGB mode to Lab mode, and then extract the L channel as the grayscale image for training; the image size of the training set is 256×256 pixels.

[0016] Preferably, the generation model in step B uses a U-net network architecture, which is composed of an encoder and a decoder, each of which has 8 modules. Among them, the encoder performs a downsampling operation, and each of its modules includes a convolution layer Conv2D and a batch normalization layer BatchNorm2d, and the activation function uses the Leaky-ReLU function for activation; each module of the decoder customizes a deconvolution layer ConvTranspose2d, the convolution kernel size of the convolution layer is (2, 4), and the deconvolution layer includes an upsampling layer and a convolution layer Conv2D with a step size of 2, and the convolution layer Conv2D uses a ReLU activation function. The L channel vector of the grayscale image is input into the generator model as a conditional vector, and the target tensor of the Lab three channels is obtained through U-net network convolution, pooling, feature fusion, deconvolution and other operations, and finally the generated image is obtained through conversion.

[0017] Preferably, the pre-training process in step B is as follows:

[0018] (1) Build U-net with ResNet backbone, load the pre-trained weights of ResNet18 architecture, and cut the model to remove the last two layers, which are GlobalAveragePooling and a linear layer for ImageNet classification task;

[0019] (2) DynamicUnet uses the Resnet skeleton to build a U-net with the required output channels and an input size of 256 pixels as the initial generator, where the output channels are 2 L and ab;

[0020] (3) Input the data of the L channel of the grayscale image into the above initial generator, calculate the loss L1(G) of the generator, and return the gradient to zero;

[0021] (4) Back propagation calculates the gradient value of each parameter and performs a step parameter update through gradient descent;

[0022] In each training, the loss function is continuously updated, the training is repeated 20 times, and the trained network is used as the generator.

[0023] Preferably, the specific process of step C is:

[0024] C1. Fix the generator and train and optimize the discriminator: Input the real image and the fake image generated by the generator to the discriminator respectively, calculate the discriminant loss through the actual output and the target output, and update the discriminator parameters;

[0025] C2. Fix the discriminator and train the optimized generator: Input the L channel of the training image in the training set into the generator, generate a fake image of the same size as the training image, input the fake image into the discriminator, calculate the generation loss through the actual output and the target output, and update the generator parameters;

[0026] C3. Repeat steps C1 and C2, and after updating the training set 20 times, the trained model can realize automatic colorization of grayscale images.

[0027] Preferably, in step C1, the number of encoder convolution kernels in the generating network and the number of convolution kernels in the discriminant network are both set to 64, the size is 4, and the epoch is set to 3-5.

[0028] Preferably, in step C, the loss function of the generative adversarial network architecture is calculated using the loss function of the conditional generative adversarial network model, the generator loss function, and the discriminator loss function to obtain the final loss function based on the generative adversarial network model; the objective function of the conditional generative adversarial network model is:

[0029] L GAN (G,D)=E x,y [log D(x,y)]+E x,z [log(1-D(x,G(x,z)))],

[0030] Among them, L GAN (G, D) represents the degree difference between the real image and the generated false image, E represents the mathematical expectation, log D(x, y) represents the probability that the discriminator determines the real image as real data, G(x, z) represents the generated false image, D(x, G(x, z)) represents the discrimination result of the discriminant model on the generated false image, and log(1-D(x, G(x, z))) represents the probability that the discriminant model determines the generated false image as a false picture;

[0031] The training of the generative model minimizes the objective function, while the discriminative model maximizes the objective function through continuous iteration, that is:

[0032] G*=arg min max L GAN (G,D)+λL1(G)

[0033] At the same time, combined with the L1 loss function, the low-frequency information of the image can be reconstructed, the high-frequency information of the image can be better preserved, and an ideal image can be generated;

[0034] Where max L GAN (G, D) means that the generative model is kept unchanged when training the discriminative model, so that the discriminative model can accurately distinguish the generated false images, that is, the value of 1- D(x, G(x, z)) is close to 1, maximizing E x,y The value of [log D(x, y)];

[0035] min L GAN (G, D) means that the discriminant model part remains unchanged when training the generative model, so that the generated false image can pass the judgment of the discriminant model, that is, the value of D(x, G(x, z)) is close to 1 and E is minimized. x,z The value of [log(1-D(x, G(x, z)))];

[0036] Where λ is the weight coefficient, which is used to adjust the weight of the loss function L1(G) of the generative model to obtain a better output image. L1(G) represents the distance between the real image and the generated image, which is defined as:

[0037] L1(G)=E x,y,z [||yG (x,y) ||1]

[0038] In the formula, G(x, z) represents the generated image, y represents the real image, and L1 represents the information reconstruction process between the real image and the generated image.

[0039] The technical solution of the present invention can produce the following technical effects:

[0040] 1. The present invention has greatly improved the optimization of training methods and training data sets through the supervised pre-training method and process of the generator, which can greatly shorten the training time on the basis of ensuring the training quality and the generalization quality of the final generated image. It is very flexible and can adapt to different usage requirements at any time. Without reducing the quality of the generated image, the selected data set can be changed and the pre-training can be quickly completed and put into use.

[0041] 2. The grayscale image colorization method based on generative adversarial network of the present invention utilizes the idea of ​​U-Net, is trained and tested on the COCO dataset, and obtains relatively accurate color images. It can greatly reduce the disadvantages of traditional methods that require manual intervention and are difficult to perform fine coloring work at the pixel level of large-size images. It can provide a new way for image colorization, produce more vibrant and realistic colors, restore more image details and make the image colors richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The generation model system architecture diagram of the present invention;

[0043] Figure 2 is a training flow chart of the present invention;

[0044] Figure 3 The structure diagram of the Resnet18 structure of the present invention after the last two layers are removed;

[0045] Figure 4 A comparison diagram of some colorization results in an embodiment of the present invention;

[0046] Wherein a is a grayscale image, b is a generated image output by the method of the present invention, and c is an original image;

[0047] Figure 5 This is a loss function trend diagram of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0049] The present invention provides a technical solution: a grayscale image colorization method based on a generative adversarial network, comprising the following steps:

[0050] A. Select a quantitative color image group from the COCO image dataset, perform decolorization, and make it into a training set;

[0051] B. Constructing a generative adversarial network architecture, including a generator model and a discriminator model, wherein the generator model is pre-trained in the generative adversarial network;

[0052] C. Input the training set obtained in step A into the generative adversarial network architecture in turn for model training and adjust the parameters to achieve convergence;

[0053] D. Preprocess the image to be processed and input the adversarial model obtained in step C to automatically colorize the grayscale image.

[0054] In the present invention, a subset of the COCO dataset is selected in step A, from which 8,000 images are selected as an optimized training set, 2,000 images are selected as a validation set, and 2,000 images are selected as a test set.

[0055] In the present invention, the decolorization method of the training set in step A is to convert the color mode of the image from RGB to Lab mode, and then extract the L channel as the grayscale image for training; the image size of the training set is 256×256 pixels.

[0056] In the present invention, the generation model in step B adopts the U-net network architecture, which consists of an encoder and a decoder, and each of the encoder and the decoder has 8 modules. Each module of the encoder includes a convolution layer Conv2D and a batch normalization layer BatchNorm2d, and the activation function uses the Leaky-ReLU function for activation. Each module of the decoder customizes a deconvolution layer ConvTranspose2d, the convolution kernel size of the convolution layer is (2, 4), the deconvolution layer includes an upsampling layer and a convolution layer Conv2D with a step size of 2, and the convolution layer Conv2D uses a ReLU activation function. The L channel vector of the grayscale image is input into the generator model as a conditional vector, and the target tensor of the Lab three-channel is obtained through U-net network convolution, pooling, feature fusion, deconvolution and other operations, and finally the generated image is obtained through conversion.

[0057] In the present invention, the pre-training process in step B is as follows:

[0058] (1) Build U-net with the ResNet backbone, load the pre-trained weights of the ResNet18 architecture, and cut the model to remove the last two layers, as shown in Figure 3 As shown, the last two layers are GlobalAveragePooling and a linear layer for the ImageNet classification task;

[0059] (2) DynamicUnet uses the Resnet skeleton to build a U-net with the required output channels and an input size of 256 pixels as the initial generator;

[0060] (3) Input the data of the L channel of the grayscale image into the initial generator, calculate the loss L1(G) of the generator, and return the gradient to zero. In this algorithm, opt uses the Adam optimizer, and lr takes 1E-4. Its main function is to make the training converge, improve the computational efficiency, and reduce the memory requirement.

[0061] (4) Back propagation calculates the gradient value of each parameter and performs a step parameter update through gradient descent;

[0062] In each training, the loss function is continuously updated, the training is repeated 20 times, and the trained network is used as the generator.

[0063] In the present invention, the specific process of step C is:

[0064] C1. Fix the generator and train and optimize the discriminator: Input the real image and the fake image generated by the generator to the discriminator respectively. The class label of the real image sample set is 1 by default, and the class label of the fake image sample set is 0. Calculate the discriminant loss through the actual output and the target output, and update the discriminator parameters.

[0065] C2. Fix the discriminator and train the optimized generator: Input the L channel of the training image in the training set into the generator, generate a fake image of the same size as the training image, input the fake image into the discriminator, calculate the generation loss through the actual output and the target output, and update the generator parameters;

[0066] C3. Repeat steps C1 and C2, and after updating the training set 20 times, the trained model can realize automatic colorization of grayscale images.

[0067] In the present invention, the number of encoder convolution kernels in the generation network and the number of convolution kernels in the discriminant network in step C1 are both set to 64, the size is 4, and the epoch is set to 3-5.

[0068] In the present invention, in step C, the loss function of the conditional generative adversarial network model, the generator loss function, and the discriminator loss function are used to calculate the loss function of the generative adversarial network architecture to obtain the final loss function based on the generative adversarial network model; the objective function of the conditional generative adversarial network model is:

[0069] L GAN (G,D)=E x,y [log D(x,y)]+E x,z [log(1-D(x,G(x,z)))],

[0070] Among them, L GAN (G, D) represents the degree difference between the real image and the generated false image, E represents the mathematical expectation, log D(x, y) represents the probability that the discriminator determines the real image as real data, G(x, z) represents the generated false image, D(x, G(x, z)) represents the discrimination result of the discriminant model on the generated false image, and log(1-D(x, G(x, z))) represents the probability that the discriminant model determines the generated false image as a false image;

[0071] The training of the generative model minimizes the objective function, while the discriminative model maximizes the objective function through continuous iteration, that is:

[0072] G*=arg min max L GAN (G,D)+λL1(G)

[0073] At the same time, combined with the L1 loss function, the low-frequency information of the image can be reconstructed, the high-frequency information of the image can be better preserved, and an ideal image can be generated;

[0074] Where max L GAN (G, D) means that the generative model is kept unchanged when training the discriminative model, so that the discriminative model can accurately distinguish the generated false images, that is, the value of 1- D(x, G(x, z)) is close to 1, maximizing E x,y The value of [log D(x, y)];

[0075] min L GAN (G, D) means that the discriminant model part remains unchanged when training the generative model, so that the generated false image can pass the judgment of the discriminant model, that is, the value of D(x, G(x, z)) is close to 1 and E is minimized. x,z The value of [log(1-D(x, G(x, z)))];

[0076] Where λ is the weight coefficient, which is used to adjust the weight of the loss function L1(G) of the generative model to obtain a better output image. L1(G) represents the distance between the real image and the generated image, which is defined as:

[0077] L1(G)=E x,y,z [||yG (x,y) ||1]

[0078] In the formula, G(x, z) represents the generated image, y represents the real image, and L1 represents the information reconstruction process between the real image and the generated image.

[0079] The effects of the present invention are further described below in conjunction with simulation experiments.

[0080] 1. Experimental conditions

[0081] The algorithm in this section is based on the caffe deep learning framework. The hardware configuration of the experiment is: processor: Intel Core i5-8500 CPU @ 3.00GHz×6; memory: 15.6GiB; GPU: Ge Force GTX 1080.

[0082] 2. Simulation content and results

[0083] The present invention selects 8000 pictures from a subset of the COCO data set as an optimized training set, 2000 pictures as a verification set, and 2000 pictures as a test set, then preprocesses the training set pictures, converts the color mode of the image from RGB to Lab mode, and then constructs a generative adversarial network model, wherein the generator model is formed after pre-training using a Resnet18 backbone network, and the number of pre-training times is 20 times; then the pre-processed training set is input into the generative adversarial network model, and the model is trained alternately, firstly the generator is fixed, the discriminator is trained and optimized, then the discriminator is fixed, the generator is trained and optimized, and after 20 iterative updates, the trained generative adversarial network model can realize automatic colorization of grayscale images. Figure 4 This is a comparison chart of the colorization results of the output image. Figure 5 This is a loss function trend diagram of the present invention.

[0084] The present invention establishes a model for colorization of grayscale images based on a generative adversarial network, wherein the generative model is pre-trained. Through a supervised pre-training method and process, the training method of the generator and the optimization of the training data set are greatly improved. Then, the training set is input, and the generative model and the discriminative model of the generative adversarial network are alternately trained to generate more accurate color images. The method of the present invention can greatly shorten the training time on the basis of ensuring the generalization quality of the training quality and the final generated image, adapt to different usage requirements, and effectively improve the flexibility of the model.

[0085] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A grayscale image colorization method based on a generative adversarial network, characterized in that: The specific steps are as follows: A. Select a quantitative color image group from the COCO image dataset, perform decolorization, and make it into a training set; B. Construct a generative adversarial network architecture, including a generative model and a discriminative model. The generative model is formed after pre-training of the Resnet18 backbone network; The pre-training process described in step B is as follows: (1) Build U-net with ResNet backbone, load the pre-trained weights of ResNet18 architecture, and cut the model to remove the last two layers; (2) DynamicUnet uses the Resnet skeleton to build a U-net with 2 output channels, L and ab, and an input size of 256 pixels as the initial generator; (3) Inputting the data of the L channel of the grayscale image into the initial generator, calculating the loss L1(G) of the generator, and returning the gradient to zero; (4) Back propagation calculates the gradient value of each parameter and performs a step parameter update through gradient descent; In each training, the loss function is continuously updated, and the training is repeated 20 times, and the trained network is used as the generator; C. Input the training set obtained in step A into the generative adversarial network architecture in turn for model training and adjust the parameters to achieve convergence; The specific process of step C is: C1. Fix the generator and train and optimize the discriminator: Input the real image and the fake image generated by the generator to the discriminator respectively, calculate the discriminant loss through the actual output and the target output, and update the discriminator parameters; C2. Fix the discriminator and train the optimized generator: Input the L channel of the training image in the training set into the generator, generate a fake image of the same size as the training image, input the fake image into the discriminator, calculate the generation loss through the actual output and the target output, and update the generator parameters; C3, repeating steps C1 and C2, and updating the training set 20 times, the trained model can realize automatic colorization of grayscale images; In step C, the loss function of the generative adversarial network architecture is calculated using the loss function of the conditional generative adversarial network model, the generator loss function, and the discriminator loss function to obtain the final loss function based on the generative adversarial network model; the objective function of the conditional generative adversarial network model is: L GAN (G,D)=E x,y [log D(x,y)]+E x,z [log(1-D(x,G(x,z)))], Among them, L GAN (G, D) represents the degree difference between the real image and the generated false image, E represents the mathematical expectation, logD(x, y) represents the probability that the discriminator determines the real image as real data, G(x, z) represents the generated false image, D(x, G(x, z)) represents the discrimination result of the discriminant model on the generated false image, and log(1-D(x, G(x, z))) represents the probability that the discriminant model determines the generated false image as a false picture; The generative model is trained to minimize the objective function, while the discriminative model is trained to maximize the objective function, namely: G*=arg min max L GAN (G,D)+λL1(G) Where max L GAN (G, D) means that the generative model is kept unchanged when training the discriminative model, so that the discriminative model can accurately identify the generated false images, that is, the value of 1-D(x, G(x, z)) is close to 1, maximizing E x,y The value of [log D(x, y)]; min L GAN (G, D) means that when training the generative model, the discriminative model part must remain unchanged, so that the generated false image can be judged by the discriminative model, that is, the value of D(x, G(x, z)) is close to 1, and E is minimized. x,z The value of [log(1-D(x, G(x, z)))]; Where λ is the weight coefficient, which is used to adjust the weight of the loss function L1(G) of the generative model to obtain a better output image. L1(G) represents the distance between the real image and the generated image, which is defined as: L1(G)=E x,y,z [||yG (x,y) ||1] In the formula, G(x,z) represents the generated image, y represents the real image, and L1 represents the information reconstruction process between the real image and the generated image; D. Preprocess the image to be processed and input the adversarial model obtained in step C to automatically colorize the grayscale image.

2. The grayscale image colorization method based on a generative adversarial network according to claim 1, characterized in that: In step A, a subset of the COCO dataset is selected, in which 8,000 images are selected as an optimized training set, 2,000 images are selected as a validation set, and 2,000 images are selected as a test set.

3. The grayscale image colorization method based on generative adversarial network according to claim 2, characterized in that: The decolorization method of the training set in step A is to convert the color mode of the image from RGB mode to Lab mode, and then extract the L channel as the grayscale image for training; the image size of the training set is 256×256 pixels.

4. A grayscale image colorization method based on a generative adversarial network according to any one of claims 1 to 3, characterized in that: The generation model in step B uses a U-net network architecture, which is composed of an encoder and a decoder, each of which has 8 modules; wherein the encoder performs a downsampling operation, and each of its modules includes a convolution layer Conv2D and a batch normalization layer BatchNorm2d, and the activation function uses a Leaky-ReLU function for activation; each module of the decoder customizes a deconvolution layer ConvTranspose2d, the convolution kernel size of the convolution layer is (2, 4), the deconvolution layer includes an upsampling layer and a convolution layer Conv2D with a step size of 2, and the convolution layer Conv2D uses a ReLU activation function.

5. The grayscale image colorization method based on generative adversarial network according to claim 1, characterized in that: In the step C1, the number of encoder convolution kernels in the generating network and the number of convolution kernels in the discriminant network are both set to 64, the size is 4, and the epoch is set to 3-5.