Palmprint image augmentation method and device based on multi-connection generative adversarial network

Abstract

The application relates to a palmprint image expansion method based on a multi-connection generative adversarial network, and specifically comprises the following steps: acquiring a real palmprint image and an initial generative adversarial network; mapping random generated noise into a real palmprint image feature space through an image generator to obtain a generated palmprint image; judging a difference value of the real palmprint image and the generated palmprint image through an image discriminator; optimizing and training the image discriminator and the image generator according to the difference value to obtain a trained generative adversarial network; and expanding the palmprint image according to the trained generative adversarial network. In the whole process, the palmprint image is generated based on the random generated noise, and the generative adversarial network is optimized and trained based on the difference between the real palmprint image and the generated palmprint image, so that the finally obtained trained generative adversarial network can realize effective expansion of the palmprint image, thereby supporting training of a palmprint recognition model of a small sample palmprint image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of palmprint feature recognition, and particularly relates to a palmprint image expansion method and device based on a multi-connection generative adversarial network, a computer device and a storage medium. BACKGROUND

[0002] Biometric recognition technology refers to a technology of performing identity recognition based on a biological feature. Palmprint recognition, as one of the main biometric recognition technologies, has been applied in more and more scenarios. Specifically, palmprint recognition is a biometric recognition technology of performing identity recognition by using a palm image from a finger tip to a wrist part, and has the characteristics of high recognition rate, low price of a collection device and good user acceptability, and gradually becomes a research hotspot in the field of biometric recognition.

[0003] However, due to privacy and cost considerations, there are few large-scale labeled palmprint images, which leads to palmprint recognition becoming a typical small sample image recognition problem, and restricts the application of a deep learning model in the field of palmprint recognition. Data expansion is one of the commonly used regularization methods, and can generate a certain amount of data similar to original data and sharing labels. Expanding small sample images can alleviate the problems of insufficient feature learning and overfitting caused by insufficient sample quantity from the data level.

[0004] Conventional palmprint image expansion mainly includes two types of methods based on traditional methods and deep learning. Traditional methods mainly include methods based on geometric transformation (cropping, flipping, scaling, rotating and adding noise, etc.), methods based on color space enhancement, methods based on sharpening and blurring, methods based on image fusion and random erasing, etc. Although the above methods can achieve expansion of palmprint images to a certain extent, there is still the problem of few labeled palmprint data, and they cannot support training of a palmprint recognition model for small sample images, that is, they cannot achieve effective expansion of palmprint images. SUMMARY

[0005] Therefore, it is necessary to provide an effective palmprint image expansion method, device, computer device and storage medium based on a multi-connection generative adversarial network in view of the above technical problems.

[0006] In a first aspect, the present application provides a palmprint image expansion method based on a multi-connection generative adversarial network. The method comprises:

[0007] obtaining a real palmprint image and an initial generative adversarial network, wherein the initial generative adversarial network comprises an image generator and an image discriminator;

[0008] mapping random generated noise to a real palmprint image feature space by the image generator to obtain a generated palmprint image;

[0009] determining a difference value between the real palmprint image and the generated palmprint image by the image discriminator;

[0010] optimizing and training the image discriminator and the image generator according to the difference value, to obtain a trained generative adversarial network;

[0011] performing palmprint image augmentation according to the trained generative adversarial network.

[0012] In one of the embodiments, the obtaining of the real palmprint image comprises:

[0013] obtaining an original palmprint image;

[0014] identifying different categories of images in the original palmprint image to obtain a category identification result;

[0015] dividing the original palmprint image into a plurality of group batch data according to categories according to the category identification result, and storing each group batch data into a first list and a second list corresponding to the category, wherein the data stored in the second list is data after disorder processing of the data stored in the first list;

[0016] performing format conversion on the data stored in the first list and the second list to obtain standard real palmprint images.

[0017] In one of the embodiments, the image generator uses U-Net 3+ as a backbone model.

[0018] The image generator comprises a plurality of encoders and a plurality of decoders, the plurality of encoders are used to obtain different scale features, the plurality of decoders are used to perform semantic recovery, dense connections are established between internal levels of the plurality of encoders, skip connections are established between the plurality of encoders and the plurality of decoders, and the number of the encoders is equal to the number of the decoders.

[0019] In one of the embodiments, the mapping of the randomly generated noise into the real palmprint image feature space by the image generator to obtain the generated palmprint image comprises:

[0020] caching the real palmprint image into an initial list;

[0021] performing a preset convolutional processing on the real palmprint image to obtain a first feature map, saving the first feature map to the end of the initial list to update the initial list;

[0022] performing feature extraction on the real palmprint image and the first feature map to obtain a second feature map, saving the second feature map to the end of the initial list to update the initial list again;

[0023] Constructing N noise encoder modules, an initial number of noise filters, the N corresponds to the network level channel number of the U-Net 3+;

[0024] The randomly generated noise is spliced with the features in the first feature map and the second feature map through the noise encoder module to generate spliced features;

[0025] Declare the upper layer feature, and initialize the upper layer feature as null;

[0026] Combine the upper layer feature and the spliced feature to generate an input feature pair, perform semantic recovery on the input feature pair through the decoder, and update the input feature pair layer by layer;

[0027] Convolution, Relu activation and batch normalization are performed on the upper layer feature of the updated input feature pair to obtain processed features;

[0028] Element-wise application of Tanh function to activate the processed features to obtain generated palmprint images.

[0029] In one of the embodiments, the image discriminator is constructed in the following way:

[0030] Constructing a pyramid split attention module;

[0031] According to the pyramid split attention module, an attention weighted DenseBlock is constructed;

[0032] Constructing a transition layer;

[0033] Adjusting the size of the feature map to obtain at least three different sizes of the DenseBlock;

[0034] Combining the at least three different sizes of the DenseBlock, and connecting adjacent two DenseBlocks through the transition layer to generate an image discriminator.

[0035] In one of the embodiments, the optimization and training of the image discriminator and the image generator according to the difference value to obtain a trained generative adversarial network includes:

[0036] Respectively creating an image generator network optimizer and an image discriminator network optimizer;

[0037] Optimizing the image discriminator through the image discriminator network optimizer according to the difference value, and optimizing the image generator through the image generator network optimizer according to the difference value.

[0038] In one of the embodiments, the optimizing and training the image discriminator and the image generator according to the difference value to obtain the trained generative adversarial network comprises:

[0039] image discriminator iteration and image generator optimization according to the difference value;

[0040] adversarial training of the image discriminator and the image generator, and optimization of network parameters until a target condition is reached, to obtain the trained generative adversarial network;

[0041] The target condition includes that the difference between the real palmprint image and the generated palmprint image received by the image discriminator in this round of training reaches Nash equilibrium, to obtain a minimum maximum target.

[0042] In a second aspect, the present application also provides a palmprint image expansion device based on a multi-connection generative adversarial network. The device comprises:

[0043] a data acquisition module configured to acquire real palmprint images and an initial generative adversarial network, the initial generative adversarial network comprising an image generator and an image discriminator;

[0044] a first processing module configured to map randomly generated noise into a real palmprint image feature space by the image generator to obtain a generated palmprint image;

[0045] a second processing module configured to determine a difference value between the real palmprint image and the generated palmprint image by the image discriminator;

[0046] an optimization and training module configured to optimize and train the image discriminator and the image generator according to the difference value to obtain a trained generative adversarial network;

[0047] an expansion module configured to perform palmprint image expansion according to the trained generative adversarial network.

[0048] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0049] acquiring real palmprint images and an initial generative adversarial network, the initial generative adversarial network comprising an image generator and an image discriminator;

[0050] mapping randomly generated noise into a real palmprint image feature space by the image generator to obtain a generated palmprint image;

[0051] determining a difference value between the real palmprint image and the generated palmprint image by the image discriminator;

[0052] optimizing and training the image discriminator and the image generator according to the difference value, to obtain a trained generative adversarial network;

[0053] performing palmprint image augmentation according to the trained generative adversarial network.

[0054] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the following steps:

[0055] obtaining real palmprint images and an initial generative adversarial network, the initial generative adversarial network comprising an image generator and an image discriminator;

[0056] mapping, by the image generator, randomly generated noise into a real palmprint image feature space, to obtain generated palmprint images;

[0057] judging, by the image discriminator, a difference value between the real palmprint images and the generated palmprint images;

[0058] optimizing and training the image discriminator and the image generator according to the difference value, to obtain a trained generative adversarial network;

[0059] performing palmprint image augmentation according to the trained generative adversarial network.

[0060] The above palmprint image augmentation method, device, computer device, storage medium and computer program product based on the multi-connection generative adversarial network obtain real palmprint images and an initial generative adversarial network; map, by the image generator, randomly generated noise into a real palmprint image feature space, to obtain generated palmprint images; judge, by the image discriminator, a difference value between the real palmprint images and the generated palmprint images; optimize and train the image discriminator and the image generator according to the difference value, to obtain a trained generative adversarial network; and perform palmprint image augmentation according to the trained generative adversarial network. In the whole process, palmprint images are generated based on randomly generated noise, and the generative adversarial network is optimized and trained based on the difference between real palmprint images and generated palmprint images, so that the trained generative adversarial network obtained finally can realize effective augmentation of palmprint images, thereby supporting training of a palmprint recognition model for small sample palmprint images. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 An application environment diagram of the palmprint image augmentation method based on the multi-connection generative adversarial network in one embodiment;

[0062] Figure 2A flowchart of a palmprint image augmentation method based on a multi-connection generative adversarial network in an embodiment;

[0063] Figure 3 A flowchart of a step of acquiring a real palmprint image in an embodiment;

[0064] Figure 4 A sub-flowchart of S200 in an embodiment;

[0065] Figure 5 A structural diagram of a DenseBlock combined with pyramid split attention;

[0066] Figure 6 A structural diagram of an image discriminator network;

[0067] Figure 7 A structural block diagram of a palmprint image augmentation device based on a multi-connection generative adversarial network in an embodiment;

[0068] Figure 8 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0069] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0070] The palmprint image augmentation method based on a multi-connection generative adversarial network provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 sends a palmprint image augmentation request to the server 104, and the server 104 responds to the palmprint image augmentation request, extracts the palmprint image carried in the palmprint image augmentation request, obtains the real palmprint image, and the server 104 reads the initial generative adversarial network pre-cached by itself, which includes an image generator and an image discriminator; through the image generator, the randomly generated noise is mapped into the real palmprint image feature space to obtain a generated palmprint image; through the image discriminator, the difference value between the real palmprint image and the generated palmprint image is judged; the image discriminator and the image generator are optimized and trained according to the difference value, and a trained generative adversarial network is obtained; and the palmprint image augmentation is performed according to the trained generative adversarial network. Further, the server 104 can feed back the augmented palmprint image data to the terminal 102. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0071] In one embodiment, as shown in Figure 2 , a palmprint image augmentation method based on a multi-connection generative adversarial network is provided. This method is applied to the server 104 in Figure 1 for example, which includes the following steps:

[0072] S100: Obtain a real palmprint image and an initial generative adversarial network, and the initial generative adversarial network includes an image generator and an image discriminator.

[0073] The real palmprint image refers to a palmprint image actually collected, which can be a real palmprint image collected by a palmprint image collection device. Further, the real palmprint image can be converted into a format that can be processed by a neural network, so as to be used for subsequent training and optimization of the initial generative adversarial network. The initial generative adversarial network refers to a generative adversarial network initially constructed, which belongs to a generative adversarial network that has not been trained. The initial generative adversarial network can specifically include an image generator and an image discriminator.

[0074] S200: Map the randomly generated noise into the real palmprint image feature space through the image generator to obtain a generated palmprint image.

[0075] The noise is randomly generated, which can introduce randomness. Specifically, the role of the noise here is to introduce randomness, so that the generated palmprint images have diversity and plasticity. The noise can be regarded as a potential "variable", and the image generator can learn from the noise how to generate different images. By training the image generator on different noise, diverse image generation results can be obtained. The role of introducing the features of the existing real palmprint images is to increase additional auxiliary information to help the image generator learn the distribution characteristics and structural information of the palmprint data, so that the image generator can more accurately simulate the distribution of the real data. After this processing, the generated palmprint image can be obtained, which can be significantly different from the real palmprint image. Therefore, the image generator parameters need to be continuously optimized in the subsequent processing process to adjust and train the image generator.

[0076] S300: Determine the difference value between the real palmprint image and the generated palmprint image by the image discriminator.

[0077] The image discriminator is used to evaluate the degree of realism of the image generated by the image generator, and to evaluate the loss between the image generated by the image generator and the real image. In actual application, the image pair can be sent into the image discriminator for training. The image pair can be [true, true] and [true, false] image pairs. Here, "false" refers to the generated palmprint image obtained by the image generator in S200.

[0078] S400: Optimize and train the image discriminator and the image generator according to the difference value to obtain a trained generative adversarial network.

[0079] Based on the difference value obtained in S300, the image discriminator and the image generator are continuously iteratively optimized and trained to gradually reduce the difference value, i.e. to gradually reduce the difference between the generated palmprint image obtained by the trained image generator and the real palmprint image, so that the trained image generator can generate "realistic" palmprint images. Further, in the continuous optimization and training process, the parameters of the image generator and the image discriminator can be continuously optimized, and the loss of the generated image, the image generator and the image discriminator can be observed to determine the convergence of the model during the optimization process. Generally, the maximum number of iterations is set as the stopping condition for iterative optimization and training.

[0080] S500: Perform palmprint image augmentation based on the trained generative adversarial network.

[0081] The trained generative adversarial network can generate sufficiently "realistic" palmprint images, so the trained generative adversarial network can be used to perform palmprint image augmentation.

[0082] The palmprint image expansion method based on the multi-connection generative adversarial network obtains a real palmprint image and an initial generative adversarial network; an image generator is used to map randomly generated noise to a real palmprint image feature space to obtain a generated palmprint image; an image discriminator is used to determine a difference value of the real palmprint image and the generated palmprint image; the image discriminator and the image generator are optimized and trained according to the difference value to obtain a trained generative adversarial network; and the palmprint image is expanded according to the trained generative adversarial network. In the whole process, the palmprint image is generated based on randomly generated noise, and the generative adversarial network is optimized and trained based on the difference between the real palmprint image and the generated palmprint image, so that the trained generative adversarial network obtained finally can realize effective expansion of the palmprint image, thereby supporting training of a palmprint recognition model for a small sample palmprint image.

[0083] As shown in Figure 3 S100 includes the following steps in one embodiment:

[0084] S120: Obtain an original palmprint image.

[0085] S140: Identify images of different categories in the original palmprint image to obtain a category recognition result.

[0086] The images of different categories are first identified for the obtained original palmprint image to obtain a category recognition result.

[0087] S160: According to the category recognition result, divide the original palmprint image into multiple group batch data according to categories, and store each group batch data into a first list and a second list corresponding to the category, wherein the data stored in the second list is data after disorder processing of the data stored in the first list. According to the category recognition result, divide the original text image into multiple group batch data according to categories, and then each group batch data has a corresponding first list and second list, and the two lists are respectively used to store image data under the current category and image data after data disorder processing. Specifically, B group batch data can be created, and the specific processing process includes the following steps:

[0088] Step 1: Take out image data of category i and store it in list X1;

[0089] Step 2: Disorder process the data in X1 and save it as list X2;

[0090] Step 3: Repeat step 1 and step 2 until all category data is saved to list X1 and list X2.

[0091] S180: Perform format conversion on the data stored in the first list and the second list to obtain standard real palmprint images.

[0092] The data in the first and second lists undergo format conversion to achieve standardized palmprint images. The specific processing steps include:

[0093] Step 4: Read in the image data from list X1 and list X2;

[0094] Step 5: Adjust the image data size and type;

[0095] Step 6: Calculate the mean and standard deviation of each channel by sampling, and standardize the image.

[0096] Here, the original palm print image is processed to obtain a standard, realistic palm print image, which facilitates subsequent processing of this data by the neural network.

[0097] In one embodiment, the image generator uses U-Net 3+ as the backbone model; the image generator includes multiple encoders and multiple decoders, the multiple encoders are used to acquire features at different scales, the multiple decoders are used to perform semantic recovery, dense connections are established between the internal layers of the multiple encoders, and skip connections are established between the multiple encoders and the multiple decoders, and the number of encoders and the number of decoders are equal.

[0098] In this embodiment, the image generator uses U-Net 3+ as the backbone model and includes multiple encoders and multiple decoders. The number of encoders and decoders can be the same, for example, four each. Multiple encoders are used to acquire features at different scales, and multiple decoders are used for semantic recovery. Dense connections are established between the internal layers of these multiple encoders, while skip connections are established between the multiple encoders and multiple decoders. Multi-scale gradient flow is achieved in the entire image generator through dense connections and skip connections, which alleviates the gradient vanishing phenomenon.

[0099] like Figure 4 As shown, in one embodiment, S200 includes:

[0100] S210: Cache real palm print images to the initial list.

[0101] Create a new list A out The actual palm print image is cached in the initial list. Preferably, the standard actual palm print image corresponding to data X0 is cached in list A. out middle.

[0102] S220: Perform a preset convolution process on the real palm print image to obtain the first feature map, and save the first feature map to the end of the initial list to update the initial list.

[0103] The encoder in the image generator performs preset convolution processing on the input data X0 to obtain a new feature map, and saves the new feature map to the end of the list A out . The update of the initial list is completed. The preset convolution processing is specifically a convolution operation performed according to a preset convolution kernel and a step size. In subsequent processing, an example in which the number of encoders and decoders is 4 is taken to be described in detail. In actual application, the encoder 0 performs a convolution operation on the input data x0 with a kernel size of 3*3 and a step size of 2, and outputs a feature x1 with a channel number of 64, which is saved to the end of the list A out .

[0104] S230: performing feature extraction on the real palmprint image and the first feature map to obtain a second feature map, saving the second feature map to the end of the initial list to update the initial list again.

[0105] The feature extraction is performed on the feature pair composed of the data corresponding to the real palmprint image and the first feature map to obtain a second feature map, the second feature map is saved to the end of the initial list, and the initial list is updated again. In actual application, the encoders 1-3 perform feature extraction on the input feature pair [x0, x1], and save the second feature map to the end of the list A out . Further, the encoder Encoder is composed of multiple convolution layers, and the specific operation is as follows:

[0106] Step 1: obtaining the input feature pair X = [x0, x1]

[0107] Step 2: performing a convolution operation on the input data x0 with a kernel size of 3*3 and a step size of 2, and modifying the feature data format to x0';

[0108] Step 3: after splicing the input data x1 and x0', performing a convolution operation with a kernel size of 3*3 and a step size of 1 to obtain output data o1;

[0109] Step 4: the feature [x1, o1] constitutes a list A in that records the output of each layer;

[0110] Step 5: splicing the feature data in the list A in , performing a convolution operation with a kernel size of 3*3 and a step size of 1 to obtain output data o i , which is supplemented to the end of the list A in ;

[0111] Step 6: repeating step 0 until the feature reaches the last layer of the Encoder block;

[0112] Step 7: splicing the list A inAfter the middle feature data, a convolution operation with a kernel size of 3*3 and a step size of 2 is performed, and dropout is executed to prevent network overfitting, and the output data o is obtained i Supplement to list A in End;

[0113] Step 8: list A in The last two feature maps are returned as decoder output;

[0114] S240: Build N noise encoder modules, initial noise filter number N corresponds to the network level channel number of U-Net 3+.

[0115] Build N noise encoder modules, initial noise filter number n noise . Its specific calculation process is as follows:

[0116] Step 1: Get the U-Net 3+ network level channel number corresponding to the nth noise encoder module, denoted as o.

[0117] Step 2: Set the last dimension channel number of the output noise tensor

[0118] Step 3: Modify the noise tensor dimension through linear transformation Linear(d, out).

[0119] Step 4: Repeat Error! Reference source not found. 0N times.

[0120] S250: Concatenate the randomly generated noise through the noise encoder module with the features in the first feature map and the second feature map to generate the spliced features.

[0121] After the randomly generated noise z is spliced with the features x1 in the first feature map and the second feature map through the noise encoder module, x curr is generated.

[0122] S260: Declare the upper layer feature and initialize the upper layer feature to empty value.

[0123] Declare the upper layer feature x pre , and initialize it to empty value.

[0124] S270: Combine the upper layer feature and the spliced feature to generate the input feature pair, perform semantic recovery on the input feature pair through the decoder, and update the input feature pair layer by layer.

[0125] The decoder (feature map decoder) performs image semantic recovery on the input feature pair [x pre , x curr ], and updates [x pre , xcurr ]. Where the decoder Decoder is composed of deconvolution and convolution layers, the specific operation is as follows:

[0126] Step 1: Get the input feature pair X = [x pre , x curr ], establish list A de and save x curr ;

[0127] Step 2: Perform up-sampling convolution operation on the input upper layer data x pre with kernel size 3*3 and step 1, modify the feature data format to x pre ';

[0128] Step 3: Concatenate the input data x curr and x pre ', update X curr ;

[0129] Step 4: Concatenate the output with list A de , perform convolution operation with kernel size 3*3 and step 1, and add the obtained output data to the end of list A de ;

[0130] Step 5: Repeat Error! Reference source not found. three times;

[0131] Step 6: Concatenate the features in list A de , perform up-sampling, convolution and activation operations, and perform dropout, and add the obtained output to the end of list A de ;

[0132] Step 7: Take the last two feature maps in list A de as the return value of this decoder [x pre , x curr ] output.

[0133] S280: Perform convolution, Relu activation and batch normalization operations on the updated upper layer features in the input feature pair to obtain processed features.

[0134] Perform three times of convolution, Relu activation and batch normalization operations on the updated X curr in S270 with kernel size 3*3 and step 1.

[0135] 290: Apply Tanh function to the processed features element by element to obtain the generated palmprint image.

[0136] The Tanh function (hyperbolic tangent) is applied element by element to the 0 features obtained by S280 to activate the elements to the interval (-1, 1). Specifically, when the output of the image generator network is activated by the Tanh function, a set of features with values between [-1, 1] is obtained, which can be regarded as generated image data.

[0137] In one embodiment, the image discriminator is constructed in the following way:

[0138] A pyramid split attention module is constructed; an attention weighted DenseBlock is constructed according to the pyramid split attention module; a transition layer is constructed; the size of the feature map is adjusted to obtain at least three DenseBlocks of different sizes; the at least three DenseBlocks of different sizes are combined, and adjacent two DenseBlocks are connected through the transition layer to generate the image discriminator.

[0139] The image discriminator uses DensNet as the backbone model. In the construction of the DenseBlock, a pyramid split attention module is added to capture spatial information of multiple scales in a long-distance scene. The network structure construction and feature acquisition process of the entire image discriminator include the following steps:

[0140] A. Construct a pyramid split attention module

[0141] Step 1: Divide the input feature map into S groups according to the channel;

[0142] Step 2: Perform convolution with different convolution kernel sizes k i for the i-th group;

[0143] Step 3: Repeat 0S times, and the convolution kernel size k i for each group is:

[0144] k i = 3 + 2 * (i-1)

[0145] The convolution kernel size increases sequentially to obtain receptive fields of different scales and extract information of different scales;

[0146] Step 4: Concatenate the feature maps obtained in the previous step and pass them through the SE module to extract the weight value of each group;

[0147] Step 5: Perform softmax normalization on the S groups of weighted values and weight the output of the pyramid split attention module.

[0148] B. Construct an attention weighted DenseBlock, and the module structure when the number of layers L = 4 is shown in Error! Reference source not found.

[0149] Step 1, for the input feature map, Batch Normalization, ReLU activation is performed;

[0150] Step 2, a convolution operation with a kernel size of 1*1 is performed on the 0 output;

[0151] Step 3, for the input of the l-th layer, the concat[x0, x1, …, x l-1 ] obtained by connecting the outputs of all previous layers is taken;

[0152] Step 4, repeat step 3 L times, and perform a nonlinear transformation operation h l-1 (·) on each layer concat[x0, x1, …, x l ], including Batch Normalization, ReLU activation, pyramid split attention weighting, and convolution operation with a kernel size of 1*1.

[0153] C, implement the transition layer. The transition layer is used to reduce the feature map size and connect two adjacent DenseBlocks.

[0154] Step 1, perform Batch Normalization and ReLU activation;

[0155] Step 2, perform a convolution operation with a kernel size of 1*1;

[0156] Step 3, perform a two-dimensional average pooling operation with a kernel size of 2*2 for down-sampling;

[0157] D, implement the image discriminator network. The image discriminator network includes three DenseBlocks, and the feature map sizes of each module are 32*32, 16*16, and 8*8, respectively. The number of layers L in each DenseBlock is 4. The structure of the image discriminator network is as shown in Figure 6 .

[0158] In one embodiment, the image discriminator and the image generator are optimized and trained according to the difference value, and a trained generative adversarial network is obtained, including:

[0159] An image generator network optimizer and an image discriminator network optimizer are respectively created; the image discriminator is optimized by the image discriminator network optimizer according to the difference value, and the image generator is optimized by the image generator network optimizer according to the difference value.

[0160] The image discriminator and the image generator use a double time scale update rule. The parameters of the image discriminator are fixed first, and the parameters of the image generator are updated. Then the parameters of the image generator are fixed, and the parameters of the image discriminator are updated. Corresponding optimizers can be created for the image generator and the image discriminator to optimize the image generator and the image discriminator through the respective corresponding optimizers. The optimizers of the image discriminator and the image generator can use different learning rates and different data update rules. In actual application, an image discriminator Adam optimizer can be created, the learning rate is set to 0.0004, each set of data of the data loader is forward propagated, the loss and the gradient of the model parameters are calculated, and the parameters of the image discriminator network are updated according to the gradient and the rule of the optimizer. An image generator Adam optimizer is created, the learning rate is set to 0.0001, 95% of the data uniformly sampled from the data loader is forward propagated, the loss and the gradient of the model parameters are calculated, and the parameters of the image generator network are updated according to the gradient and the rule of the optimizer.

[0161] In one embodiment, the image discriminator and the image generator are optimized and trained according to the difference value to obtain a trained generative adversarial network, which includes:

[0162] The image discriminator and the image generator are iterated and optimized according to the difference value. The image discriminator and the image generator are trained in an adversarial manner, and the network parameters are optimized until a target condition is reached to obtain a trained generative adversarial network. The target condition includes that the difference between the real palmprint image and the generated palmprint image received by the image discriminator in this round of training reaches Nash equilibrium to obtain a minimum maximum target.

[0163] In the process of optimizing and training the image discriminator and the image generator, the network parameters of the image discriminator and the image generator are constantly adjusted until a target condition is reached. The target condition refers to that the difference between the real palmprint image (real image) received by the image discriminator after this round of training and the generated palmprint image (fake image) obtained by the image generator reaches Nash equilibrium to obtain an image discriminator loss of a minimum maximum target.

[0164] To describe the above training process in detail, the entire scheme will be described in detail from the generation of palmprint image data as follows:

[0165] 1. Obtain two palmprint image data x1 and x2 of the same category from the list X1 and X2;

[0166] 2. Perform image discriminator iteration;

[0167] 2.1. Input x1 into the image generator to generate a sample

[0168] 2.2, input the real sample pair X r = [x1, x2] and the generated sample pair X into the image discriminator respectively, and calculate the difference values D(X r ) and D(X g );

[0169] 2.3, calculate the image discriminator loss L D :

[0170]

[0171] wherein, is the real sample data distribution; is the generated sample data distribution, which is implicitly defined by X g = G(z), z ~ p(z); wherein is the gradient penalty term, which is usually taken as λ = 10;

[0172] 2.4, back-propagate the loss L D , and optimize the optimizer;

[0173] 2.5, calculate the image generator loss L G

[0174]

[0175] 2.6, back-propagate the loss L G , and optimize the image generator.

[0176] 3, the image discriminator and the image generator are trained in confrontation and the network parameters are optimized, the image generator network maps the noise to the feature space, the image discriminator network receives the generated sample or the real data sample pair, and distinguishes the two, finally reaches the Nash equilibrium, and the minimum maximum target:

[0177]

[0178] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0179] Based on the same inventive concept, the embodiments of the present application also provide a multi-connection generative adversarial network based palmprint image expansion device for implementing the multi-connection generative adversarial network based palmprint image expansion method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-connection generative adversarial network based palmprint image expansion device embodiments provided below can be referred to the limitations of the multi-connection generative adversarial network based palmprint image expansion method described above, which will not be repeated here.

[0180] In one embodiment, as shown in Figure 7 a multi-connection generative adversarial network based palmprint image expansion device is provided, comprising:

[0181] The data acquisition module 100 is configured to acquire real palmprint images and an initial generative adversarial network, wherein the initial generative adversarial network comprises an image generator and an image discriminator.

[0182] The first processing module 200 is configured to map random generated noise to a real palmprint image feature space by the image generator to obtain a generated palmprint image.

[0183] The second processing module 300 is configured to determine a difference value between the real palmprint image and the generated palmprint image by the image discriminator.

[0184] The optimization and training module 400 is configured to optimize and train the image discriminator and the image generator according to the difference value to obtain a trained generative adversarial network.

[0185] The expansion module 500 is configured to perform palmprint image expansion according to the trained generative adversarial network.

[0186] In one embodiment, the data acquisition module 100 is further configured to acquire original palmprint images, identify different categories of images in the original palmprint images to obtain a category identification result, divide the original palmprint images into a plurality of group batch data according to the category identification result, and store each group batch data into a first list and a second list corresponding to the category, wherein the data stored in the second list is data after disorder processing of the data stored in the first list; and perform format conversion on the data stored in the first list and the second list to obtain standard real palmprint images.

[0187] In one embodiment, the image generator uses U-Net 3+ as a backbone model; the image generator comprises a plurality of encoders and a plurality of decoders, the plurality of encoders are configured to acquire different scale features, the plurality of decoders are configured to perform semantic recovery, dense connections are established between the internal levels of the plurality of encoders, skip connections are established between the plurality of encoders and the plurality of decoders, and the number of encoders is equal to the number of decoders.

[0188] In one of the embodiments, the first processing module 200 is further configured to cache the real palmprint image to an initial list; perform a preset convolutional processing on the real palmprint image to obtain a first feature map, save the first feature map to the end of the initial list to update the initial list; perform feature extraction on the real palmprint image and the first feature map to obtain a second feature map, save the second feature map to the end of the initial list to update the initial list again; construct N noise encoder modules, the initial number of noise filters, N corresponds to the network level channel number of U-Net 3+; splice the randomly generated noise with the features in the first feature map and the second feature map through the noise encoder module to generate spliced features; declare an upper layer feature and initialize the upper layer feature as a null value; combine the upper layer feature and the spliced features to generate an input feature pair, perform semantic recovery on the input feature pair through a decoder, and update the input feature pair layer by layer; perform convolution, Relu activation and batch normalization operations on the upper layer feature in the updated input feature pair to obtain a processed feature; apply a Tanh function to the processed feature element by element to obtain a generated palmprint image.

[0189] In one of the embodiments, the image discriminator is constructed in the following manner: a pyramid split attention module is constructed; an attention weighted DenseBlock is constructed according to the pyramid split attention module; a transition layer is constructed; the size of the feature map is adjusted to obtain at least three DenseBlocks with different sizes; at least three DenseBlocks with different sizes are combined, and adjacent two DenseBlocks are connected through the transition layer to generate the image discriminator.

[0190] In one of the embodiments, the optimization and training module 400 is further configured to create an image generator network optimizer and an image discriminator network optimizer respectively; optimize the image discriminator through the image discriminator network optimizer according to the difference value, and optimize the image generator through the image generator network optimizer according to the difference value.

[0191] In one of the embodiments, the optimization and training module 400 is further configured to perform image discriminator iteration and image generator optimization according to the difference value; perform adversarial training on the image discriminator and the image generator, and optimize the network parameters until the target condition is reached to obtain a trained generative adversarial network; wherein the target condition includes that the difference between the real palmprint image and the generated palmprint image received by the image discriminator in the current round of training reaches Nash equilibrium to obtain a minimum maximum target.

[0192] The modules in the palmprint image augmentation device based on the multi-connection generative adversarial network can be realized by software, hardware, or a combination thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the modules.

[0193] In one embodiment, a computer device, which can be a server, has an internal structure as shown in Figure 8 The computer device includes a processor, a memory, and a network interface connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data such as an initial generative adversarial network. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a palmprint image augmentation method based on a multi-connection generative adversarial network.

[0194] Those skilled in the art can understand that Figure 8 The structure shown in the above

[0195] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the palmprint image augmentation method based on the multi-connection generative adversarial network.

[0196] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the palmprint image augmentation method based on the multi-connection generative adversarial network.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0199] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0200] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A palmprint image augmentation method based on a multi-connection generative adversarial network, characterized in that, The method includes: Acquire a real palm print image and an initial generative adversarial network, wherein the initial generative adversarial network includes an image generator and an image discriminator; The image generator maps randomly generated noise into the feature space of a real palm print image to obtain a generated palm print image. The image discriminator determines the difference between the real palm print image and the generated palm print image. The image discriminator and the image generator are optimized and trained based on the difference value to obtain a trained generative adversarial network. Palmprint image augmentation is performed based on the trained generative adversarial network. The image generator uses U-Net 3+ as the backbone model; the image generator includes multiple encoders and multiple decoders, the multiple encoders are used to acquire features at different scales, the multiple decoders are used to perform semantic recovery, dense connections are established between the internal layers of the multiple encoders, and skip connections are established between the multiple encoders and the multiple decoders, and the number of encoders is equal to the number of decoders; The image discriminator is constructed as follows: a pyramid segmentation attention module is constructed; based on the pyramid segmentation attention module, an attention-weighted DenseBlock is constructed; a transition layer is constructed; the size of the feature map is adjusted to obtain at least three DenseBlocks of different sizes; the at least three DenseBlocks of different sizes are combined, and adjacent DenseBlocks are connected through the transition layer to generate the image discriminator; the convolution kernels in the pyramid segmentation attention module are sequentially increased to obtain receptive fields of different scales and extract information at different scales; The step of mapping randomly generated noise into the feature space of a real palmprint image through the image generator to obtain a generated palmprint image includes: caching the real palmprint image to an initial list; performing a preset convolution process on the real palmprint image to obtain a first feature map, and saving the first feature map to the end of the initial list to update the initial list; extracting features from the real palmprint image and the first feature map to obtain a second feature map, and saving the second feature map to the end of the initial list to update the initial list again; constructing N noise encoder modules, with an initial number of noise filters, where N is related to the U-Net. The network has 3+ layers and channels. Randomly generated noise is concatenated with features in the first and second feature maps using a noise encoder module to generate concatenated features. Upper-layer features are declared and initialized to null values. The upper-layer features and the concatenated features are combined to generate input feature pairs. The input feature pairs are semantically restored using a decoder, and the input feature pairs are updated layer by layer. Convolution, ReLU activation, and batch normalization operations are performed on the upper-layer features in the updated input feature pairs to obtain processed features. The Tanh function is applied element-wise to activate the processed features to generate a palmprint image. The construction of N noise encoder modules, with an initial number of noise filters, includes: Step 1: Obtaining the number of U-Net 3+ network layer channels corresponding to the nth noise encoder module, denoted as... Step 2: Set the number of channels in the last dimension of the output noise tensor. Step 3: Through linear transformation (Linear( , Modify the noise tensor dimension; Step 4: Repeat the process 0 to 0 N times.

2. The method according to claim 1, characterized in that, The process of obtaining the real palm print image includes: Obtain the original palm print image; Different categories of images are identified in the original palm print image to obtain category identification results; Based on the category recognition results, the original palm print image is divided into multiple batches of data according to the category. Each batch of data is stored in the first list and the second list corresponding to the category. The data stored in the second list is the data after the data stored in the first list has been shuffled. The data stored in the first list and the second list are converted to obtain a standard real palm print image.

3. The method according to claim 1, characterized in that, The step of optimizing and training the image discriminator and the image generator based on the difference value to obtain the trained generative adversarial network includes: Create separate image generator network optimizers and image discriminator network optimizers; The image discriminator is optimized by the image discriminator network optimizer based on the difference value, and the image generator is optimized by the image generator network optimizer based on the difference value.

4. The method according to claim 1, characterized in that, The step of optimizing and training the image discriminator and the image generator based on the difference value to obtain the trained generative adversarial network includes: The image discriminator is iterated and the image generator is optimized based on the difference values; The image discriminator and image generator are subjected to adversarial training, and the network parameters are optimized until the target conditions are met, resulting in a trained generative adversarial network. The target condition includes the difference between the real palm print image and the generated palm print image received by the image discriminator in this round of training reaching Nash equilibrium, thus obtaining the minimum-maximum target.

5. A palmprint image augmentation device based on a multi-connection generative adversarial network, characterized in that, The device includes: The data acquisition module is used to acquire real palm print images and an initial generative adversarial network, which includes an image generator and an image discriminator. The first processing module is used to map randomly generated noise into the feature space of a real palm print image through the image generator to obtain a generated palm print image; The second processing module is used to determine the difference between the real palm print image and the generated palm print image through the image discriminator; An optimization and training module is used to optimize and train the image discriminator and the image generator based on the difference value to obtain a trained generative adversarial network. An augmentation module is used to augment palmprint images based on the trained generative adversarial network. The image generator uses U-Net 3+ as the backbone model; the image generator includes multiple encoders and multiple decoders, the multiple encoders are used to acquire features at different scales, the multiple decoders are used to perform semantic recovery, dense connections are established between the internal layers of the multiple encoders, and skip connections are established between the multiple encoders and the multiple decoders, and the number of encoders is equal to the number of decoders; The image discriminator is constructed as follows: a pyramid segmentation attention module is constructed; based on the pyramid segmentation attention module, an attention-weighted DenseBlock is constructed; a transition layer is constructed; the size of the feature map is adjusted to obtain at least three DenseBlocks of different sizes; the at least three DenseBlocks of different sizes are combined, and adjacent DenseBlocks are connected through the transition layer to generate the image discriminator; the convolution kernels in the pyramid segmentation attention module are sequentially increased to obtain receptive fields of different scales and extract information at different scales; The first processing module is further configured to: cache real palmprint images to an initial list; perform preset convolution processing on the real palmprint images to obtain a first feature map, and save the first feature map to the end of the initial list to update the initial list; extract features from the real palmprint images and the first feature map to obtain a second feature map, and save the second feature map to the end of the initial list to update the initial list again; construct N noise encoder modules, with an initial number of noise filters, where N corresponds to the number of network layer channels in the U-Net 3+; concatenate randomly generated noise with features in the first and second feature maps through the noise encoder modules to generate concatenated features; declare upper-layer features and initialize the upper-layer features to null values; combine the upper-layer features and the concatenated features to generate input feature pairs, perform semantic recovery on the input feature pairs through a decoder, and update the input feature pairs layer by layer; perform convolution, ReLU activation, and batch normalization operations on the upper-layer features in the updated input feature pairs to obtain processed features; and apply the Tanh function element by element to activate the processed features to obtain a generated palmprint image. The construction of N noise encoder modules, with an initial number of noise filters, includes: Step 1: Obtaining the number of U-Net 3+ network layer channels corresponding to the nth noise encoder module, denoted as... Step 2: Set the number of channels in the last dimension of the output noise tensor. Step 3: Through linear transformation (Linear( , Modify the noise tensor dimension; Step 4: Repeat the process 0 to 0 N times.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images