A method for extracting contours of optical footprint images based on GAN network

Abstract

The present invention relates to a method for extracting contours of an optical red footprint image based on a GAN network, comprising: collecting original optical red footprint images through an optical footprint collector; making training sets and test sets; constructing a generator through a residual network; constructing a discriminator through a PatchGAN Markov discriminator; forming a CycleGAN cyclic generative adversarial network with the generator and the discriminator as a training network; sending the training set into the training network for training; using the trained generator as a contour extraction test network, inputting source domain data of the test set, and obtaining the contour of the optical red footprint image. The present invention can directly perform a set of unified process operations on the original optical red footprint image that has been simply preprocessed to extract the contour image, thereby simplifying the contour extraction process; the present invention can ignore the differences between different original images, use a network with the same parameters to process the same batch of data, and reduce the amount of calculation for contour extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and contour extraction, and in particular to a method for extracting the contour of an optical naked footprint image based on a GAN network. Background Art

[0002] The study of footprints is not limited to the field of criminal investigation, but also involves all aspects of our lives. For example, by studying the differences in human footprints when doing different sports, we can improve shoemaking technology and develop high-tech sports shoes suitable for different sports to better protect our bodies during exercise. In the medical field, medical experts can explore the relationship between footprints and certain diseases by studying the impact of certain diseases on the footprints produced by patients during walking, and use this to predict the type and stage of the patient's disease and improve the speed of diagnosis.

[0003] The morphological characteristics of the contour of optical naked footprint images are a very important feature in the study of naked footprints. In the current research process, the contour is usually extracted from the naked footprint image by first median filtering and then binarization, which is usually called the traditional method. The median filter processes the gray value of each pixel in the image into the median of the gray values ​​of all pixels in a neighborhood window of the pixel, thereby eliminating isolated noise points and erasing the texture information inside the optical naked footprint. Binarization sets the gray value of the pixel on the image to 0 or 255, and the entire image presents an obvious black and white effect, thereby reducing the amount of data in the image and highlighting the contour information of the naked footprint in the image. However, the traditional method has high requirements for the quality of the original optical naked footprint image. If there is a large area of ​​block noise in the original image or there is a break in the arch area of ​​the footprint, it is impossible to use the traditional method to extract a highly reliable contour image. However, the quality of the footprint images obtained at the crime scene is often not high. Summary of the invention

[0004] The purpose of the present invention is to provide an optical bare footprint image contour extraction method based on a GAN network, which can process block noise in the original image, complete the incomplete footprint in the original image, and more flexibly generate a contour image with higher credibility.

[0005] To achieve the above object, the present invention adopts the following technical solution: a method for extracting the contour of an optical naked footprint image based on a GAN network, the method comprising the following steps in order:

[0006] (1) collecting original optical footprint images by an optical footprint collector;

[0007] (2) Create training sets and test sets;

[0008] (3) Constructing the generator through the residual network;

[0009] (4) Construct the discriminator through PatchGAN Markov discriminator;

[0010] (5) The generator and the discriminator form a CycleGAN cyclic generative adversarial network, and the CycleGAN cyclic generative adversarial network is used as the training network;

[0011] (6) Send the training set to the training network for training;

[0012] (7) Use the trained generator as a contour extraction test network, input the source domain data of the test set, and obtain the optical footprint image contour.

[0013] The step (2) specifically comprises the following steps:

[0014] (2a) All original optical barefoot images are unified into the right foot image and the ruler in the image is erased;

[0015] (2b) Randomly extract images from the original optical footprint image as the source domain data of the training set and the test set, and then randomly extract images and process them as the target domain data of the training set.

[0016] The step (3) specifically comprises the following steps:

[0017] (3a) The generator is a residual network structure with 9 residual blocks, c7s1_k represents a downsampling layer with k filters, a convolution kernel size of 7*7, and a step size of 1; dk represents a downsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 2; rk represents a residual block containing two convolution layers with k filters and a convolution kernel size of 3*3; uk represents an upsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 1 / 2; the structure of the generator is represented as: c7s1_64, d128, d256, r256, r256, r256, r256, r256, r256, r256, r256, u128, u64, c7s1_3;

[0018] A self-attention module is added after the two convolutional layers of the last residual block of the generator to recalibrate the obtained features. The formula is as follows:

[0019]

[0020]

[0021] Where: e(x i ) is the i-th row in the feature map e(x) in space e, g(xj ) represents the jth row in the feature map g(x) in space g, s i，j represents the element in the i-th row and j-th column of the two-dimensional matrix s, β i，j is the element in the i-th row and j-th column of the two-dimensional matrix β, i represents the number of rows, N represents the maximum value range of i, h(x i ) is the i-th row in the feature map h(x) in space h; Formula (1) is the attention mask calculation formula, and Formula (2) is the feature recalibration calculation formula;

[0022] (3b) Perform spectral normalization after each convolutional layer in the generator:

[0023] f w (x)=Wx=U∑Vx (3)

[0024] Among them, U represents an m*m orthogonal matrix, ∑ is an m*n diagonal matrix, the numbers on the diagonal are the singular values ​​of the weight matrix W, V is an n*n orthogonal matrix, and x is a variable.

[0025] The step (5) specifically comprises the following steps:

[0026] Use a i and b i Representing a sample in the source domain and the target domain respectively, the CycleGAN cycle generation adversarial network is represented as follows:

[0027] a i →G_A(a i )→G_B(G_A(a i ))

[0028] b i →G_B(b i )→G_A(G_B(b i ))

[0029] A sample a in source domain A i After being transformed into the target domain by the generator G_A, it is marked as G_A(a i ), then G_A(a i ) is transformed into the source domain by the generator G_B, marked as G_B(G_A(a i ));

[0030] A sample b in the target domain B i After being transformed into the source domain by the generator G_B, it is marked as G_B(b i ), then G_B(b i ) is transformed into the target domain by the generator G_A, marked as G_A(G_B(b i));

[0031] G_A(a i ) and G_B(b i ) are all generated samples, G_B(G_A(a i )) and G_A(G_B(b i )) are all cyclically generated samples;

[0032] (5a) Use Wasserstein distance instead of the traditional adversarial loss based on cross entropy. The calculation formula of Wasserstein distance is as follows:

[0033]

[0034] In the formula, Π(P r , P g ) represents the true distribution P r With the generated distribution P g The set of all joint distributions of the combination, γ represents any distribution in the set of joint distributions, variable x represents a real sample in γ, variable y represents a generated sample in γ, E (x，y)～γ [||xy||] represents the expectation of γ for the distance between x and y, and calculates the lower bound of the expectation in the set of joint distributions.

[0035] In order to add Wasserstein distance to the training process of CycleGAN, Formula (4) is transformed into Formula (5):

[0036]

[0037] In the formula, ||f|| L≤K It means that the function f is Lipschitz continuous on K, that is, the absolute value of the derivative function of f is less than K, K ≥ 0, sup means taking the upper bound of the function f that satisfies the condition;

[0038] In the training process of CycleGAN, the function f(x) is defined as an objective function containing weight w for solution, that is, formula (5) is converted into formula (6):

[0039]

[0040] Optimize the objective function f w (x) is to find the value that makes f w The optimal weight w value of (x) is to find the optimal solution of w, and define the function f(x) as an objective function f containing the weight w. w (x), due to f w(x) = Wx, W is the weight matrix, w is the weight; || f w || L≤K Represents the function f w is Lipschitz continuous on K, i.e. f w The absolute value of the derivative function is less than K; E represents the expectation, P r Represents the true distribution, x~P r Indicates that the sample x comes from the true distribution, P g represents the generated distribution, x~P g Indicates that the sample x comes from the generating distribution, Indicates that the sample x comes from the true distribution P r Time w The expected value of (x), Indicates that sample x comes from the generating distribution P g Time w (x) The expected value;

[0041] According to formula (6), the objective function of the discriminator in the CycleGAN cycle generative adversarial network is reconstructed as formula (7), while ensuring that the function f w (x) Under the premise of Lipschitz continuity, optimizing the discriminator is equivalent to optimizing the Wasserstein distance between the true distribution and the generated distribution:

[0042]

[0043] D(x) is the objective function of the discriminator, which is equal to the Wasserstein distance between the true distribution and the generated distribution;

[0044] After introducing the Wasserstein distance, the adversarial loss of the generator is shown in formula (8), and the adversarial loss of the discriminator is shown in formula (9):

[0045]

[0046]

[0047] In the formula, G represents the generator, D represents the discriminator, N represents the maximum value range of i, and a i represents a source domain sample in N pairs of samples, b i Represents a target domain sample;

[0048] (5b) The cycle-consistent loss is introduced to limit the one-to-one correspondence between the original image and the contour image. The cycle-consistent loss is shown as follows:

[0049]

[0050] Among them, ||G_B(G_A(a i ))-a i ||1To find G_B(G_A(a i )) and a i Manhattan distance of i Represents a source domain sample, which passes through generator G_A and then generator G_B to obtain a new image G_B in the source domain (G_A(a i )); b i represents a target domain sample, G_A(G_B(b i )) represents the new image obtained by bi passing through the generator twice.

[0051] The step (6) specifically comprises the following steps:

[0052] (6a) Setting network hyperparameters, taking a sample from the source domain and target domain of the training set respectively, and generating the corresponding generated sample and cyclic generated sample through the CycleGAN cyclic generative adversarial network;

[0053] (6b) Calculate the adversarial loss lossD_A, lossD_B of the discriminator;

[0054] (6c) Calculate the adversarial loss lossG_A, lossG_B and cycle consistency loss lossC of the generator, that is, the total loss of the generator is as follows:

[0055] lossG total =lossG_A+lossG_B+lossC

[0056] (6d) Select the RMSProp optimizer, set the optimizer parameters, train and save the CycleGAN cyclic generative adversarial network.

[0057] It can be seen from the above technical solutions that the beneficial effects of the present invention are as follows: first, the present invention can directly perform a set of unified process operations on the original optical footprint image that has been simply preprocessed to extract the contour image, thereby simplifying the contour extraction process; second, the present invention can ignore the differences between different original images, use a network with the same parameters to process the same batch of data, and reduce the amount of calculation for contour extraction; third, the present invention uses a more flexible GAN network algorithm, which can weaken the correspondence between the pixel values ​​of the same position pixel points in the original image and the contour image, ignore the defects in the original image, and handle some situations that cannot be handled by traditional methods; fourth, the present invention adds a self-attention mechanism to the CycleGAN structure, which can enable the entire network to learn complete structural information based on the connection between the structures in the image, dominate the direction of learning, and improve the effect of contour extraction; fifth, the present invention adds a spectral normalization operation to the CycleGAN structure, and uses the Wasserstein distance instead of the traditional adversarial loss based on the cross entropy, which can effectively avoid the problem of gradient disappearance in the GAN network during training, making the training of the GAN network more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of the method of the present invention;

[0059] Figure 2 This is a schematic diagram of the fourth generation optical footprint collector;

[0060] Figure 3 is the original image;

[0061] Figure 4 is the preprocessed image;

[0062] Figure 5 is the contour image;

[0063] Figure 6 Schematic diagram of the self-attention mechanism structure;

[0064] Figure 7 Schematic diagram of the CycleGAN structure;

[0065] Figure 8 Schematic diagram of cycle consistency loss. DETAILED DESCRIPTION

[0066] like Figure 1 As shown, a method for extracting the contour of an optical footprint image based on a GAN network includes the following steps in sequence:

[0067] (1) collecting original optical footprint images by an optical footprint collector;

[0068] (2) Create training sets and test sets;

[0069] (3) Constructing the generator through the residual network;

[0070] (4) Construct the discriminator through PatchGAN Markov discriminator;

[0071] (5) The generator and the discriminator form a CycleGAN cyclic generative adversarial network, and the CycleGAN cyclic generative adversarial network is used as the training network;

[0072] (6) Send the training set to the training network for training;

[0073] (7) Use the trained generator as a contour extraction test network, input the source domain data of the test set, and obtain the optical footprint image contour.

[0074] The step (2) specifically comprises the following steps:

[0075] (2a) All original optical barefoot images are unified into the right foot image and the ruler in the image is erased;

[0076] (2b) Randomly extract images from the original optical footprint image as the source domain data of the training set and the test set, and then randomly extract images and process them as the target domain data of the training set.

[0077] The step (3) specifically comprises the following steps:

[0078] (3a) The generator is a residual network structure with 9 residual blocks, c7s1_k represents a downsampling layer with k filters, a convolution kernel size of 7*7, and a step size of 1; dk represents a downsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 2; rk represents a residual block containing two convolution layers with k filters and a convolution kernel size of 3*3; uk represents an upsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 1 / 2; the structure of the generator is represented as: c7s1_64, d128, d256, r256, r256, r256, r256, r256, r256, r256, r256, u128, u64, c7s1_3;

[0079] A self-attention module is added after the two convolutional layers of the last residual block of the generator to recalibrate the obtained features. The formula is as follows:

[0080]

[0081]

[0082] Where: e(x i ) is the i-th row in the feature map e(x) in space e, g(x j ) represents the jth row in the feature map g(x) in space g, s i，j represents the element in the i-th row and j-th column of the two-dimensional matrix s, β i，j is the element in the i-th row and j-th column of the two-dimensional matrix β, i represents the number of rows, N represents the maximum value range of i, h(x i ) is the i-th row in the feature map h(x) in space h; Formula (1) is the attention mask calculation formula, and Formula (2) is the feature recalibration calculation formula;

[0083] (3b) Perform spectral normalization after each convolutional layer in the generator:

[0084] f w (x)=Wx=U∑Vx (3)

[0085] Among them, U represents an m*m orthogonal matrix, ∑ is an m*n diagonal matrix, the numbers on the diagonal are the singular values ​​of the weight matrix W, V is an n*n orthogonal matrix, and x is a variable.

[0086] The step (5) specifically comprises the following steps:

[0087] Use a i and b i Representing a sample in the source domain and the target domain respectively, the CycleGAN cycle generation adversarial network is represented as follows:

[0088] a i →G_A(a i )→G_B(G_A(a i ))

[0089] b i →G_B(b i )→G_A(G_B(b i ))

[0090] A sample a in source domain A i After being transformed into the target domain by the generator G_A, it is marked as G_A(a i ), then G_A(a i ) is transformed into the source domain by the generator G_B, marked as G_B(G_A(a i ));

[0091] A sample b in the target domain B i After being transformed into the source domain by the generator G_B, it is marked as G_B(b i ), then G_B(bi ) is transformed into the target domain by the generator G_A, marked as G_A(G_B(b i ));

[0092] G_A(a i ) and G_B(b i ) are all generated samples, G_B(G_A(a i )) and G_A(G_B(b i )) are all cyclically generated samples;

[0093] (5a) Use Wasserstein distance instead of the traditional adversarial loss based on cross entropy. The calculation formula of Wasserstein distance is as follows:

[0094]

[0095] In the formula, Π(P r , P g ) represents the true distribution P r With the generated distribution P g The set of all joint distributions of the combination, γ represents any distribution in the set of joint distributions, variable x represents a real sample in γ, variable y represents a generated sample in γ, E (x，y)～γ [||xy||] represents the expectation of γ for the distance between x and y, and calculates the lower bound of the expectation in the set of joint distributions.

[0096] In order to add Wasserstein distance to the training process of CycleGAN, Formula (4) is transformed into Formula (5):

[0097]

[0098] In the formula, ||f|| L≤K It means that the function f is Lipschitz continuous on K, that is, the absolute value of the derivative function of f is less than K, K ≥ 0, sup means taking the upper bound of the function f that satisfies the condition;

[0099] In the training process of CycleGAN, the function f(x) is defined as an objective function containing weight w for solution, that is, formula (5) is converted into formula (6):

[0100]

[0101] Optimize the objective function f w (x) is to find the value that makes f wThe optimal weight w value of (x) is to find the optimal solution of w, and define the function f(x) as an objective function f containing the weight w. w (x), due to f w (x) = Wx, W is the weight matrix, w is the weight; || f w || L≤K Represents the function f w is Lipschitz continuous on K, i.e. f w The absolute value of the derivative function is less than K; E represents the expectation, P r Represents the true distribution, x~P r Indicates that the sample x comes from the true distribution, P g represents the generated distribution, x~P g Indicates that the sample x comes from the generating distribution, Indicates that the sample x comes from the true distribution P r Time w The expected value of (x), Indicates that sample x comes from the generating distribution P g Time w (x) The expected value;

[0102] According to formula (6), the objective function of the discriminator in the CycleGAN cycle generative adversarial network is reconstructed as formula (7), while ensuring that the function f w (x) Under the premise of Lipschitz continuity, optimizing the discriminator is equivalent to optimizing the Wasserstein distance between the true distribution and the generated distribution:

[0103]

[0104] D(x) is the objective function of the discriminator, which is equal to the Wasserstein distance between the true distribution and the generated distribution;

[0105] After introducing the Wasserstein distance, the adversarial loss of the generator is shown in formula (8), and the adversarial loss of the discriminator is shown in formula (9):

[0106]

[0107]

[0108] In the formula, G represents the generator, D represents the discriminator, N represents the maximum value range of i, and a i represents a source domain sample in N pairs of samples, b i Represents a target domain sample;

[0109] (5b) The cycle-consistent loss is introduced to limit the one-to-one correspondence between the original image and the contour image. The cycle-consistent loss is shown as follows:

[0110]

[0111] Among them, ||G_B(G_A(a i ))-a i ||1To find G_B(G_A(a i )) and a i Manhattan distance of i Represents a source domain sample, which passes through generator G_A and then generator G_B to obtain a new image G_B in the source domain (G_A(a i )); b i represents a target domain sample, G_A(G_B(b i )) represents the new image obtained by bi passing through the generator twice.

[0112] The step (6) specifically comprises the following steps:

[0113] (6a) Setting network hyperparameters, taking a sample from the source domain and target domain of the training set respectively, and generating the corresponding generated sample and cyclic generated sample through the CycleGAN cyclic generative adversarial network;

[0114] (6b) Calculate the adversarial loss lossD_A, lossD_B of the discriminator;

[0115] (6c) Calculate the adversarial loss lossG_A, lossG_B and cycle consistency loss lossC of the generator, that is, the total loss of the generator is as follows:

[0116] lossG total =lossG_A+lossG_B+lossC

[0117] (6d) Select the RMSProp optimizer, set the optimizer parameters, train and save the CycleGAN cyclic generative adversarial network.

[0118] Embodiment 1

[0119] Step 1: Collecting optical red footprint images through an optical footprint collector;

[0120] The optical footprint images were collected using the fourth generation optical footprint collector (FMC500IV) developed by Hangzhou Chuangheng Electronic Technology Development Co., Ltd. Figure 2As shown in the figure, the effective collection area of ​​the device is the black part on the front of the instrument. Connect the power interface and USB interface on the left and right sides of the instrument according to the instructions, and install the corresponding driver and computer software on the computer to collect. When the footprint contacts the collection surface, the prism below the device collection surface will undergo total reflection, and the captured image will be transmitted to the computer software interface and saved locally on the computer. The device can collect one left or right foot optical barefoot image each time, and store the collected image data according to the unified naming specification. The resolution of the stored image data is 325*670.

[0121] Step 2: Preprocess the collected optical naked footprint images and create a data set. In step 1, 300 optical naked footprint images with different IDs were collected. Each ID contains 10 left foot images and 10 right foot images, a total of 6000 images. The original images are as follows: Figure 3 shown.

[0122] Due to the setting of the acquisition instrument, in the original optical naked-footprint image, there is a ruler used to mark the length on the thumb side of the footprint, such as Figure 3 By detecting the direction of the ruler in the image, first convert all left foot images into right foot images, then set all pixel values ​​of the 35-pixel portion on the left side of the image to 255 to erase the ruler, and finally convert the image to black and white to obtain the processed image data as shown in Figure 4 shown.

[0123] 450 different images were randomly selected and divided into 3 groups, each with 150 images. The first group was defined as the source domain data of the training set, the second group was defined as the source domain data of the test set, and the third group was manually processed into the target domain data (i.e., optical footprint contour images, such as Figure 5 As shown), it is defined as the target domain data of the training set.

[0124] Step 3: Define the generator. The basic structure of the generator used in the present invention is a ResNet structure with 9 residual blocks. The ResNet structure adds residual units through a short-circuit mechanism, which can well solve the degradation problem of deep networks. The basic structure of the generator of the present invention can be expressed as: c7s1_64, d128, d256, r256, r256, r256, r256, r256, r256, r256, r256, u128, u64, c7s1_3.

[0125] In order to enable the network to focus on the correlation between bare footprint structures and improve the quality of generated contours, the present invention adds a self-attention module after the two Convolution layers of the last residual block of the generator to recalibrate the obtained features.

[0126] The self-attention module can focus on the correlation between points in the feature map, that is, calculate the influence of other points on the current point, and then obtain the structural features of the image, improving the network's ability to depict image details. The detailed structure diagram of the self-attention module is as follows Figure 6 In order to reduce the amount of network computation, the feature map x (with dimensions of H*W*C) is first subjected to three 1*1 convolutions to obtain three feature spaces e, g and h. Then, the feature maps of the e, g and h spaces are expanded into HW*C. The rows of the expanded feature maps represent all channel information at a certain position before expansion, and the columns represent all position information at a certain channel before expansion. The feature map in space e is transposed and multiplied with the feature map in space g and normalized by the softmax layer to obtain the attention mask, as shown in formula (1), where β i，j It represents the influence of the jth position on the ith position in the image. Finally, the feature map of space h is multiplied by the attention mask, as shown in formula (2), so as to increase the response of the key area and reduce the response of other areas. Then the size is restored to obtain the output feature map y (dimension is H*W*C). The self-attention module enables the entire network to learn complete structural information based on the connection between the structures in the image, that is, to learn from local information to global information, avoid blind learning of the network, dominate the direction of learning, and improve network performance.

[0127] In order to solve the problem that the CycleGAN cyclic generative adversarial network is unstable and prone to gradient disappearance during training, the present invention subsequently introduces the Wasserstein distance to replace the traditional adversarial loss based on cross entropy. Therefore, the present invention performs a spectral normalization operation after each Convolution layer in the generator (before the self-attention module when it involves the self-attention module) to provide the conditions for the Wasserstein distance to hold, that is, the function satisfies 1-Lipschitz continuity, which means that the maximum gradient of any point of the function is 1.

[0128] The spectral normalization operation can convert f w The weight of (x) is limited to a certain range to ensure that the function f w (x) satisfies 1-Lipschitz continuity. As shown in formula (3), for the function f w The weight matrix W of (x) m*nPerform singular value decomposition (SVD). After decomposing the weight matrix W, U represents an m*m orthogonal matrix, ∑ is an m*n diagonal matrix, the numbers on the diagonal are the singular values ​​of the weight matrix W, and V is an n*n orthogonal matrix. Through SVD decomposition, the complex matrix can be decomposed into three simple matrix multiplications. The three matrices represent the three matrix transformations of W, where U and V represent rotation transformations, and ∑ represents stretching transformations. Only the stretching process will change the modulus of the vector, so the function f w The 1-Lipschitz continuity of (x) is only related to ∑. Define the maximum singular value obtained after the matrix SVD decomposition as the spectral coefficient of the matrix. Then, during the training process, the weight matrix W is divided by its spectral coefficient after each convolution operation to ensure that the stretched matrix ∑ after SVD decomposition is less than 1, that is, to ensure that the function f w (x) satisfies 1-Lipschitz continuity. The above process is the spectral normalization operation of the weight matrix W.

[0129] Step 4: Define the discriminator. The basic structure of the discriminator used in the present invention is PatchGAN. This structure is used to determine whether a patch of a certain size covered by an image comes from the original image. Compared with the global discriminator, such a patch-level discriminator has fewer parameters and can process images of any size in a fully convolutional manner. If cxsi_k represents a Convolution-LeakyReLU layer with k filters, a convolution kernel size of x*x, and a step size of i; cksi represents a Convolution-InstanceNorm-LeakyReLU layer with k filters, a convolution kernel size of 4*4, and a step size of i, where the slope of LeakyReLU is 0.2. The basic structure of the discriminator of the present invention can be expressed as: c4s2_64, c128s2, c256s2, c512s1, c4s1_1. The LeakyReLU layer is deleted from the last layer.

[0130] For the same reasons as in step 3.1 and step 3.2, a spectral normalization operation is performed after each Convolution layer in the discriminator, and a self-attention module is added after the Convolution layer in c512s1 (after the spectral normalization operation).

[0131] Step 5: Build a training network. The training network model of the present invention refers to the CycleGAN structure, such as Figure 7As shown in the figure, this structure can well complete the image translation work in the absence of paired data sets. Where A represents the source domain of the training set, that is, the optical bare footprint image dataset, and B represents the target domain of the training set, that is, the contour dataset; G_A and G_B are the generator structures defined in step 3, G_A is used to generate images in the target domain from images in the source domain, and G_B is used to generate images in the source domain from images in the target domain; D_A and D_B are the discriminator structures defined in step 4, D_A is used to determine whether the image is a real image in the target domain, and D_B is used to determine whether the image is an image in the source domain; G_A and D_A form a forward GAN network, and G_B and D_B form a reverse GAN network. The forward GAN network and adversarial loss can ensure the generation of sufficiently realistic target domain images; the reverse GAN network and cycle consistency loss are added to limit the strict one-to-one correspondence between the target domain image obtained by translating the source domain image and the source image in the process of image translation.

[0132] Since the adversarial loss based on cross entropy in traditional GAN ​​networks has certain defects, problems such as gradient vanishing and mode collapse may occur during training. The present invention uses Wasserstein distance to replace the traditional adversarial loss based on cross entropy, which simplifies the structure of the loss function while ensuring stable training.

[0133] Compared with the adversarial loss based on cross entropy, Wasserstein distance will not suffer from mode collapse due to the lack of intersection between distributions. When two distributions do not intersect, the adversarial loss based on cross entropy is abrupt and cannot provide gradient information, while Wasserstein distance is smooth and can provide gradient information. In order to add Wasserstein distance to the training process of GAN, first transform formula (4) into formula (5), where ||f|| L≤K It means that the function f is Lipschitz continuous on K, that is, the absolute value of the derivative function f does not exceed K (K ≥ 0), and sup means taking the upper bound of the function f that satisfies the condition.

[0134] In the training process of CycleGAN, the function f(x) is defined as an objective function containing weight w for solution, that is, formula (5) is converted into formula (6). Since the validity of formula (6) must ensure that the function f w (x) is 1-Lipschitz continuous on K, so it is necessary to use spectral normalization during training to restrict f w The weight w in (x) is within a certain range, then f w The derivative of (x) with respect to x will be restricted to a certain range.

[0135] According to formula (6), the objective function of the discriminator in the CycleGAN cycle generative adversarial network can be reconstructed as formula (7), while ensuring that the function f w (x) Under the premise of Lipschitz continuity, optimizing the discriminator is equivalent to optimizing the Wasserstein distance between the true distribution and the generated distribution. Therefore, the gradient disappearance problem does not occur.

[0136] After the introduction of Wasserstein distance, the adversarial loss of the generator of the present invention is shown in formula (8), and the adversarial loss of the discriminator is shown in formula (9). Where G represents the generator, D represents the discriminator, N represents the number of samples in one round of training, and a i represents a source domain sample among N samples, b i Represents a target domain sample.

[0137] The CycleGAN structure believes that the learned function should have cycle consistency, such as Figure 8 As shown, for each image a in the data domain A, in the circular translation, it should be possible to return a to the origin of the translation, so as to ensure that the result of the image translation has a one-to-one correspondence with the original image, and vice versa. In other words:

[0138] a i →G_A(a i )→G_B(G_A(a i ))≈a i

[0139] b i →G_B(b i )→G_A(G_B(b i ))≈b i

[0140] Thus, the cycle consistency loss is introduced, as shown in formula (6).

[0141] Step 6: Send the data set produced in step 2.2 into the built training network for training.

[0142] Set the number of network training epochs to 200, batchsize = 1, the source domain and source domain samples are represented by A and a respectively, and the target domain and target domain samples are represented by B and b respectively. That is, the process of translating one sample twice is as follows:

[0143] real_a→fake_b→rec_a

[0144] real_b→fake_a→rec_b

[0145] Take out a sample real_a and real_b from the source domain and the target domain respectively. Real_a is generated by the generator G_A to generate the generated sample fake_b in the target domain, and then fake_b is generated by the generator G_B to generate the generated sample rec_a in the source domain; the generation process of real_b→fake_a→rec_b is similar.

[0146] According to formula (9), the loss of discriminator D_B is calculated by real_a and fake_a as lossD_B; the loss of discriminator D_A is calculated by real_b and fake_b as lossD_A.

[0147] According to formula (8), the loss of generator G_A is calculated by fake_b as lossG_A; the loss of generator G_B is calculated by fake_a as lossG_B. According to formula (10), the cycle consistency loss is calculated by real_a, rec_a, real_b, rec_b as lossC. That is, the total loss of the generator is:

[0148] lossG total =lossG_A+lossG_B+lossC

[0149] The initial learning rate is set to 0.002. A fixed learning rate is used in the first 100 epochs, and the learning rate is linearly decayed to 0 in the next 100 epochs. The RMSProp optimizer is used to optimize the discriminator D_A parameters according to lossD_A, the discriminator D_B parameters according to lossD_B, and the discriminator D_B parameters according to lossG total Optimize the parameters of generators G_A and G_B and save the trained network.

[0150] Step 7: Use the trained generator G_A to build a contour extraction test network, input the source domain data of the test dataset, output the obtained target domain data, and test the network.

[0151] In summary, the present invention can directly perform a set of unified process operations on the original optical footprint image that has been simply preprocessed to extract the contour image, thereby simplifying the contour extraction process; the present invention can ignore the differences between different original images, use a network with the same parameters to process the same batch of data, and reduce the amount of calculation for contour extraction; the present invention uses a more flexible GAN network algorithm, which can weaken the correspondence between the pixel values ​​of the pixel points at the same position in the original image and the contour image, ignore the defects in the original image, and handle some situations that cannot be handled by some traditional methods.

Claims

1. A method for extracting contours of optical red footprint images based on a GAN network, characterized in that: The method comprises the following steps in order: (1) collecting original optical footprint images by an optical footprint collector; (2) Create training sets and test sets; (3) Constructing the generator through the residual network; (4) Construct the discriminator through PatchGAN Markov discriminator; (5) The generator and the discriminator are combined into a CycleGAN cyclic generative adversarial network, and the CycleGAN cyclic generative adversarial network is used as the training network; (6) Send the training set to the training network for training; (7) Using the trained generator as a contour extraction test network, inputting the source domain data of the test set, and obtaining the optical footprint image contour; The step (5) specifically comprises the following steps: Use a i and b i Representing a sample in the source domain and the target domain respectively, the CycleGAN cycle generation adversarial network is represented as follows: a i →G_A(a i )→G_B(G_A(a i )) b i →G_B(b i )→G_A(G_B(b i )) A sample a in source domain A i After being transformed into the target domain by the generator G_A, it is marked as G_A(a i ), then G_A(a i ) is transformed into the source domain by the generator G_B, marked as G_B(G_A(a i )); A sample b in the target domain B i After being transformed into the source domain by the generator G_B, it is marked as G_B(b i ), then G_B(b i ) is transformed into the target domain by the generator G_A, marked as G_A(G_B(b i )); G_A(a i ) and G_B(b i ) are all generated samples, G_B(G_A(a i )) and G_A(G_B(b i )) are all cyclically generated samples; (5a) Using Wasserstein distance instead of the traditional adversarial loss based on cross entropy, (5b) Cycle consistency loss is introduced to restrict the one-to-one correspondence between the original image and the contour image.

2. According to the GAN network-based optical red footprint image contour extraction method of claim 1, it is characterized in that: The step (2) specifically comprises the following steps: (2a) All original optical barefoot images are unified into the right foot image and the ruler in the image is erased; (2b) Randomly extract images from the original optical footprint image as the source domain data of the training set and the test set, and then randomly extract images after processing as the target domain data of the training set.

3. The optical red footprint image contour extraction method based on the GAN network according to claim 1 is characterized in that: The step (3) specifically comprises the following steps: (3a) The generator is a residual network structure with 9 residual blocks, c7s1_k represents a downsampling layer with k filters, a convolution kernel size of 7*7, and a step size of 1; dk represents a downsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 2; rk represents a residual block containing two convolution layers with k filters and a convolution kernel size of 3*3; uk represents an upsampling layer with k filters, a convolution kernel size of 3*3, and a step size of 1 / 2; the structure of the generator is represented as: c7s1_64, d128, d256, r256, r256, r256, r256, r256, r256, r256, r256, u128, u64, c7s1_3; A self-attention module is added after the two convolutional layers of the last residual block of the generator to recalibrate the obtained features. The formula is as follows: Where: e(x i ) is the i-th row in the feature map e(x) in space e, g(x j ) represents the jth row in the feature map g(x) in space g, s i，j represents the element in the i-th row and j-th column of the two-dimensional matrix s, β i，j is the element in the i-th row and j-th column of the two-dimensional matrix β, i represents the number of rows, N represents the maximum value range of i, h(x i ) is the i-th row in the feature map h(x) in space h; Formula (1) is the attention mask calculation formula, and Formula (2) is the feature recalibration calculation formula; (3b) Perform spectral normalization after each convolutional layer in the generator: f w (x)=Wx=UΣVx (3) Among them, U represents an m*m orthogonal matrix, ∑ is an m*n diagonal matrix, the numbers on the diagonal are the singular values ​​of the weight matrix W, V is an n*n orthogonal matrix, and x is a variable.

4. The optical red footprint image contour extraction method based on the GAN network according to claim 1 is characterized in that: The calculation formula of the Wasserstein distance is as follows: In the formula, Π(P r , P g ) represents the true distribution P r With the generated distribution P g The set of all joint distributions of the combination, γ represents any distribution in the set of joint distributions, variable x represents a real sample in γ, variable y represents a generated sample in γ, E (x，y)～γ [||xy||] represents the expectation of γ for the distance between x and y, and calculates the lower bound of the expectation in the set of joint distributions. In order to add Wasserstein distance to the training process of CycleGAN, formula (4) is transformed into formula (5): In the formula, ||f|| L≤K It means that the function f is Lipschitz continuous on K, that is, the absolute value of the derivative function of f is less than K, K ≥ 0, sup means taking the upper bound of the function f that satisfies the condition; In the training process of CycleGAN, the function f(x) is defined as an objective function containing weight w for solution, that is, formula (5) is converted into formula (6): Optimize the objective function f w (x) is to find the value that makes f w The optimal weight w value of (x) is to find the optimal solution of w, and define the function f(x) as an objective function f containing the weight w. w (x), due to f w (x) = Wx, W is the weight matrix, w is the weight; || f w || L≤K Represents the function f w is Lipschitz continuous on K, i.e. f w The absolute value of the derivative function is less than K; E represents the expectation, P r Represents the true distribution, x~P r Indicates that the sample x comes from the true distribution, P g represents the generated distribution, x~P g Indicates that the sample x comes from the generating distribution, Indicates that the sample x comes from the true distribution P r Time w The expected value of (x), Indicates that sample x comes from the generating distribution P g Time w (x) The expected value; According to formula (6), the objective function of the discriminator in the CycleGAN cycle generative adversarial network is reconstructed as formula (7), while ensuring that the function f w (x) Under the premise of Lipschitz continuity, optimizing the discriminator is equivalent to optimizing the Wasserstein distance between the true distribution and the generated distribution: D(x) is the objective function of the discriminator, which is equal to the Wasserstein distance between the true distribution and the generated distribution; After introducing the Wasserstein distance, the adversarial loss of the generator is shown in formula (8), and the adversarial loss of the discriminator is shown in formula (9): In the formula, G represents the generator, D represents the discriminator, N represents the maximum value range of i, and a i represents a source domain sample in N pairs of samples, b i Represents a target domain sample; The cycle-consistent loss is shown as follows: Among them, ||G_B(G_A(a i ))-a i ||1To find G_B(G_A(a i )) and a i Manhattan distance of i Represents a source domain sample, which passes through generator G_A and then generator G_B to obtain a new image G_B in the source domain (G_A(a i )); b i represents a target domain sample, G_A(G_B(b i )) represents the new image obtained by bi passing through the generator twice.

5. The method for extracting optical red footprint image contour based on GAN network according to claim 1, characterized in that: The step (6) specifically comprises the following steps: (6a) Setting network hyperparameters, taking a sample from the source domain and target domain of the training set respectively, and generating the corresponding generated sample and cyclic generated sample through the CyoleGAN cyclic generative adversarial network; (6b) Calculate the adversarial loss lossD_A, lossD_B of the discriminator; (6c) Calculate the adversarial loss lossG_A, lossG_B and cycle consistency loss lossC of the generator, that is, the total loss of the generator is as follows: lossG total =lossG_A+lossG_B+lossC (6d) Select the RMSProp optimizer, set the optimizer parameters, train and save the CycleGAN cyclic generative adversarial network.

Images