Prior-guided Iris Image Restoration System
By generating an iris image repair system for adversarial networks, the problem of degradation of recognition accuracy caused by low resolution and blurred iris images is solved, and clear iris images are generated, improving the accuracy and robustness of iris recognition.
Patent Information
- Application Number
- CN202211074175.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-02
- Publication Date
- 2025-07-29
- Estimated Expiration
- 2042-09-02
AI Technical Summary
Existing iris recognition systems are susceptible to interference during image acquisition in medium and long distances and depth of field, resulting in low resolution and blurred iris images, affecting the accuracy of iris recognition.
Iris image repair system based on generative adversarial networks is adopted, including degradation removal subnetwork, prior estimation subnetwork and prior fusion subnetwork, to generate clear iris images through feature extraction and style information fusion, and to improve image quality using hourglass structure and attention mechanism.
The accuracy of iris recognition is improved, and the generated images are consistent with real images in pixel-level consistency and high-level semantics, enhancing the robustness of iris recognition.
Smart Images
Figure CN115424337B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and specifically, to an iris image restoration system based on prior guidance. Background Art
[0002] As one of the most potential biometric recognition technologies, iris recognition has been widely applied in fields and industries such as public security and justice, financial banks, mobile Internet, etc. due to its advantages such as high accuracy and non-contact. However, existing iris recognition systems are vulnerable to external factors during image acquisition. On the one hand, in medium and long-distance scenarios (such as distances above 1 meter), the acquisition resolution of images decreases, making it difficult to capture high-quality iris images with sufficient resolution. On the other hand, the depth of field of iris cameras is small (i.e., the range of clear imaging is small). When the user is outside the depth of field or is moving, the image is prone to blurring.
[0003] Low-resolution and blurred low-quality iris images will lead to a reduction in the effective recognition area of the iris, a reduction in effective texture information, and at the same time will affect the accuracy of iris preprocessing such as iris segmentation, thereby leading to a decrease in the recognition accuracy. Summary of the Invention
[0004] The present invention proposes an iris image restoration system based on prior guidance, which solves the problem of the decrease in iris recognition accuracy caused by low-resolution and blurred iris images in related technologies.
[0005] The technical solution of the present invention is as follows: It includes a generator and a discriminator. The generator includes:
[0006] A degradation removal sub-network for extracting features from the input image to obtain a first image y c ; The first image y c is a super-resolution and de-blurred image;
[0007] A prior estimation sub-network for obtaining the style information p in the prior knowledge according to the first image y c ; The style information p i contains the color, shape, and texture information of the iris image; i
[0008] A prior fusion sub-network for fusing the first image y c and the style information p i to obtain a prior fusion feature; The prior fusion feature is used to generate a clear restored image
[0009] Further, the prior estimation sub-network includes a plurality of residual channel attention modules connected in a hourglass structure, a skip connection mechanism is adopted between symmetric layers of the hourglass structure, and a convolutional layer is added between the symmetric layers; any one of the symmetric layers includes: two residual channel attention modules symmetrically distributed on both sides of the center of the hourglass structure;
[0010] The last N residual channel attention modules in the prior estimation sub-network are connected to the prior fusion sub-network, and the style information p i includes the output of the last N residual channel attention modules; N is a positive integer.
[0011] Further, the prior fusion sub-network adopts a channel-aware attention mechanism;
[0012] In the hourglass structure of the prior estimation sub-network, in the order from front to back, the output of the nth module among the last N residual channel attention modules is expressed as; n is a natural number, and n ≤ N;
[0013] When fusing the first image y c and the style information p i to obtain the prior fusion feature, the prior fusion sub-network is specifically used for:
[0014] Performing N times of feature fusion operations to obtain the prior fusion feature;
[0015] Among them, the nth feature fusion operation includes:
[0016] Using convolution and a mapping function to perform an affine transformation on p in to convert p in into the weight coefficient in the channel attention mechanism
[0017] Splitting the feature map into spatial features with spatial information and spatial features
[0018] According to the weight coefficient and the spatial feature to obtain the style feature;
[0019] Concatenating the style feature and the spatial feature to obtain the fusion feature corresponding to the nth feature fusion operation;
[0020] In the first feature fusion operation, the feature map is the feature obtained by downsampling the first image y c ;
[0021] In other secondary feature fusion operations, the feature map is the fusion feature corresponding to the previous feature fusion operation;
[0022] The fusion feature obtained after the Nth feature fusion operation is used as the prior fusion feature.
[0023] Furthermore, the degradation removal sub-network specifically includes: a convolutional layer, three residual channel attention modules, and two convolutional layers connected in sequence. The convolutional layers all use a 3×3 convolutional kernel. Furthermore, the loss function of the generator is:
[0024]
[0025] where L rec 、L perc 、L style 、 are the reconstruction loss, perceptual loss, style loss, and adversarial loss respectively, and λ1, λ2, λ3, and λ4 are trade-off parameters respectively;
[0026] The reconstruction loss y is the real image, and ||||1 represents the L1 norm;
[0027] The perceptual loss The perceptual loss is calculated by inputting the restored image and the real image y into the VGG-19 network. Φ i is the activation map of the ith layer of the VGG-19 network; N i is the total number of pixels in the activation map of the ith layer;
[0028] The style loss is the C j ×C j Gram matrix constructed from the activation map of the ith layer, and C j is the number of channels of the activation map of the ith layer;
[0029] The adversarial loss D Ra is the discriminator, represents the error between the real image y and the restored image and E[] is the expected value calculation.
[0030] The working principle and beneficial effects of the present invention are:
[0031] Based on the adversarial generative network, the present invention first uses the degradation removal sub-network to roughly restore the input image, which can reduce the prediction difficulty of prior knowledge; the first image y of the rough restoration cIt is sent to two branches, namely the prior estimation sub-network and the prior fusion sub-network. The prior estimation sub-network extracts features and estimates the style information p in the iris prior knowledge. i , the style information p i often contains information such as the color, shape, and texture of the iris image; at the same time, the first image y c is input into the prior fusion sub-network and fused with the style information p i to enable the style information p i to guide feature extraction, which is beneficial to generating a clear restored image, thereby ensuring the accuracy of iris recognition. Brief Description of the Drawings
[0032] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments.
[0033] Figure 1 is a schematic diagram of the overall network structure of the generator of the present invention;
[0034] Figure 2 is a schematic diagram of the prior fusion sub-network structure in the present invention;
[0035] Figure 3 is a schematic diagram of the data set of this embodiment (CA4L);
[0036] Figure 4 is a schematic diagram of the data set of this embodiment (CA4T);
[0037] Figure 5 is the cropped iris image. Detailed Description of the Embodiments
[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of the present invention.
[0039] 1. Overall Framework
[0040] As Figure 1 shown, this embodiment is based on a generative adversarial network, including two parts: a generator and a discriminator. The discriminator uses a relative average discriminator. The generator mainly consists of three parts: a degradation removal sub-network, a prior fusion sub-network, and a prior estimation sub-network. Assume that x is a degraded low-resolution and blurred input image, and y is a real image. q and p are the output high-quality iris image and the prior information predicted by the prior estimation sub-network, respectively. The degraded image x (input image) may be too blurred to estimate prior knowledge. Therefore, in this embodiment, the degradation removal sub-network is first used to extract features and recover a rough super-resolution, de-blurred image:
[0041] y c = l(x) (1)
[0042] l represents the mapping function from the input image x to the high-resolution image y c Then, y c is input into the prior estimation sub-network P and the prior fusion sub-network F, respectively:
[0043] p = P(y c ) (2)
[0044]
[0045] Among them, the prior fusion sub-network F extracts features from y c and performs soft fusion of the prior knowledge obtained by the prior estimation sub-network P and the extracted features based on the attention mechanism to generate a clearer high-quality iris image. Send it to the discriminator to judge true or false, and play a game with the generator, forcing the generator to generate realistic images that can deceive the discriminator.
[0046] 2. Network Structure Details
[0047] This part describes the structure and operation details of the degradation removal sub-network, the prior fusion sub-network, and the prior estimation sub-network of the generator.
[0048] 2.1 Degradation Removal Sub-network
[0049] As Figure 1 shown, first, the degradation removal sub-network is used to roughly recover the degraded image. The degraded image may lead to inaccurate prior information estimation due to unknown severe degradation. Therefore, using the degradation removal sub-network for simple adaptive recovery can reduce the prediction difficulty of prior knowledge. This sub-network first uses a 3×3 convolutional layer to convert a 3-channel image into a 64-channel feature map, followed by a stack of 3 residual networks and two convolutional layers, all using 3×3 convolutional kernels and a channel number of 64.
[0050] 2.2 Prior Estimation Sub-network
[0051] The roughly restored iris images are sent to two branches, namely the prior estimation sub-network and the prior fusion sub-network. The prior estimation sub-network extracts features and estimates the iris prior information. The iris has both a unique topological structure (such as a circular structure, etc.) and unique texture distribution characteristics (such as random distribution, circumferential correlation, radial extensibility, etc.), and its prior knowledge can be modeled and estimated. However, due to the complexity of iris texture, simple key points, parsing diagrams, etc. are difficult to provide effective gains for image restoration. Considering that the style information of the image often contains a large amount of information such as the color, shape, and texture of the image, style priors at different stages and scales can be used to guide feature extraction.
[0052] Specifically, in this embodiment, a hourglass structure is adopted to estimate the prior information. The detailed structure of the prior estimation network is as Figure 1 shown, which is stacked by residual channel attention modules of different scales. In order to effectively integrate cross-scale features and retain spatial information at different scales, the hourglass structure uses a skip connection mechanism between symmetric layers, and then uses a 3×3 convolutional layer to post-process the obtained features. Finally, the outputs p i1 、p i2 、p i3 (p i1 、p i2 、p i3 arranged in the order from input to output) are used as the style information of the iris image. The prior features of the three parts are different due to their different positions and scales in the network. Among them, the smaller and earlier parts tend to be information such as color and shape, while the later parts tend to be texture information.
[0053] 2.3 Prior Fusion Sub-network
[0054] The prior fusion sub-network is also composed of residual channel attention module groups of different scales, as Figure 1 shown. Through scale changes, richer features are extracted from the input rough restored image . Let the output of the prior estimation network be p i , and y c obtains a feature map after feature extraction It is necessary to fuse the input prior knowledge and the feature map. Existing fusion methods usually directly splice or add the prior knowledge and the features, such as in the face restoration task FSRNet, etc. However, simple operations may ignore the priority of specific information existing in the prior knowledge, resulting in the prior knowledge not being fully utilized.
[0055] Therefore, in this embodiment, a new perceptual attention fusion module is used to solve the above problems. For the outputs p i1, p i2 , p i3 , perform the feature fusion operation three times to obtain the prior fusion feature; the first feature fusion operation is as Figure 2 shown, including:
[0056] Perform an affine transformation on p i1 using convolution and a mapping function, and transform p i1 into the weight coefficient in the channel attention mechanism At the same time, evenly divide the feature map into features with spatial information and feature Then the style feature is:
[0057]
[0058] Finally, concatenate the style feature with the spatial feature to obtain the fusion feature corresponding to the first feature fusion operation Since the weight coefficient is added to the style feature it ensures the full utilization of the style information p i1 .
[0059] In the other feature fusion operations, use the feature after the previous fusion as the feature map
[0060] The feature after the third fusion is used as the prior fusion feature.
[0061] 2.4 Loss Function
[0062] In this embodiment, perceptual loss, style loss, reconstruction loss, and adversarial loss are used to constrain the quality of the image generated by the generator. The reconstruction loss is used to measure the pixel-level difference between images, and simultaneously supervise the roughly restored intermediate image and the finally output image. The formula is as follows:
[0063]
[0064] The generative adversarial network GAN has shown strong capabilities in image inpainting and can generate more realistic images than deep models based on L1 and L2 losses. Its key idea is to use a discriminative network to distinguish between the reconstructed image and the real image, and train the generative network to deceive the discriminator.
[0065] Let the discriminator be D Ra , and the discriminator loss is defined as:
[0066]
[0067] The adversarial loss of the generator is in a symmetric form:
[0068]
[0069] To capture high-level semantics and simulate human perception of images, a perceptual loss is proposed. By using a network pre-trained on the ImageNet dataset to extract iris feature maps, the differences between different features are calculated as follows:
[0070]
[0071] where Φ i is the activation map of the i-th layer of the VGG-19 backbone network. In the work of this embodiment, Φ i corresponds to the activation maps of ReLu3_1, ReLu4_1, and ReLu5_1.
[0072] The style loss is used to calculate the style differences between pictures. In this embodiment, it is mainly used to constrain the style prior, and the formula is as follows:
[0073]
[0074] is the Gram matrix C constructed from the selected activation maps j ×C j where C j is the number of channels of the activation map, and the activation maps used are the same as those in the perceptual loss.
[0075] The overall generator loss is:
[0076]
[0077] λ1, λ2, λ3, and λ4 are the trade-off parameters for the reconstruction loss, perceptual loss, style loss, and adversarial loss, respectively.
[0078] 3. Experimental verification
[0079] 3.1 Dataset
[0080] In this embodiment, the lamp dataset (CASIA Iris Image DatabaseV4 Lamp, CA4L) and the thousand-person dataset (CASIA Iris Image Database V4 Thousand, CA4T) in the Chinese Academy of Sciences iris dataset are used, as Figure 3 , Figure 4As shown. CA4L includes 822 categories, consisting of the left and right eyes of 411 individuals. The left and right eyes of the same individual are different categories, with a total of 16,212 images. CA4T includes 2,000 categories, consisting of the left and right eyes of 1,000 individuals, with a total of 20,000 pictures. The two datasets are merged and screened to obtain a merged dataset of 2,738 categories and 28,089 iris images for this experiment. The original image resolution is 640x480, and after iris detection and localization, it is cropped to a resolution of 224x224, as Figure 5 shown.
[0081] The model is mainly trained on the generated dataset, which approximates real low-quality images and is generalized to real-world images during the testing process. The following degradation model is used in this embodiment to synthesize training data:
[0082]
[0083] The high-quality image y is first convolved with the blur kernel k σ , and then a downsampling operation with a scale factor r is performed. For each training pair, the downsampling scale factor, motion blur kernel, and Gaussian blur kernel are randomly selected from {1:4}, {3:15}, {3:15} respectively in this embodiment. The test dataset is synthesized in the same way as the training data. The test data and the training dataset do not overlap.
[0084] 3.2 Experimental Settings
[0085] During the training process, the original data of the dataset is grayscale single-channel iris images, which are converted into 3-channel RGB images in this embodiment, so that the input and output of the model are RGB images with a size of 224×224×3. The number of pictures loaded in batches is 16. The learning rate is initially set to 0.0001 and decays by half every 100,000 iterations. Adam is used as the optimizer, where β1 = 0.9, β2 = 0.999, ζ = 10 -8 .
[0086] The deep learning framework used is Pytorch, and the GPU is NVIDIA GeForce RTX3090. Four loss functions are used in this embodiment, and four loss function weights are used to balance each loss function. λ1, λ2, λ3, λ4 are 1, 0.01, 0.01, 0.1 respectively.
[0087] 3.3 Comparative Analysis of Generation Metrics
[0088] First, the quality of the generated images is evaluated and analyzed in this embodiment. To more effectively evaluate the method proposed in this embodiment, this method is compared with advanced restoration algorithms proposed in the current general image restoration and face image restoration fields, including the general super-resolution algorithms EDSR, RCAN, SRGAN, ESRGAN; the face super-resolution algorithms SISN, SPARNet; and the deblurring algorithm DeblurGAN v2; and quantitative comparisons are made. The evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). For fair comparison, all of the above algorithms were retrained on the dataset provided in this embodiment. Table 1 lists the quantitative results of all methods.
[0089] It can be observed from Table 1 that the method proposed in this embodiment achieves the best PSNR, SSIM, and LPIPS performance on both datasets. It is worth noting that although the GAN-based method (DeblurGAN v2) has slightly worse PSNR and SSIM metrics, it is much better than the ordinary convolutional networks (EDSR, RCAN) in terms of LPIPS performance, indicating that the generative adversarial network can restore more image details. And the method proposed in this embodiment is excellent in each metric, indicating that this method restores more details of real images and retains more pixel-level consistency with real images compared to other methods, demonstrating the superiority of the method in this embodiment.
[0090] Table 1 Quantitative comparison results of each method
[0091]
[0092] 3.4 Ablation experiments
[0093] In this embodiment, an ablation study is further carried out to measure the effectiveness of the style loss function, the prior fusion module, and the prior estimation sub-network. First, the baseline model that removes the style loss, the prior fusion module, and the prior estimation sub-network simultaneously is called Model 1, the model that only removes the style loss while retaining the prior fusion module and the prior estimation sub-network is called Model 2, the model that only removes the prior fusion module is called Model 3, the model that removes the style loss and the prior fusion module simultaneously is Model 4, and the model that includes all modules is Model 5. The PSNR, SSIM, and LPIPS performance on the dataset is shown in Table 2.
[0094] Table 2 Ablation test results
[0095]
[0096] As can be seen from Table 2, compared with the baseline model 1, the quality of the reconstructed image of model 4 has been greatly improved, indicating that the prior knowledge guidance has a large gain for the algorithm. The performance of model 2 has been greatly improved compared with model 1 and model 4, indicating that the prior fusion module improves the utilization of prior knowledge. Model 3 has advantages compared with model 1 and model 4, indicating that the style loss strengthens the constraint on the iris style information. The method proposed in this embodiment can not only efficiently extract image features, but also introduce prior knowledge guidance and use style loss for collaborative supervision to obtain more realistic restored images. The results prove the superiority of the proposed method.
[0097] The iris image restoration system based on prior guidance proposed in this embodiment can restore iris images with unknown degradations such as low resolution, motion blur, and defocus blur, and improve the robustness of iris recognition. Based on the iris prior knowledge, the generation network can be guided. In this embodiment, the hourglass network is used to perform distribution modeling on the style information of the target to be generated to improve the quality of the generated image. Moreover, this embodiment proposes an attention fusion mechanism to integrate prior knowledge and inject style information at different scales into the generation network step by step to improve the utilization rate of prior information.
[0098] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A prior-guided iris image restoration system for restoring low-resolution blurred images, comprising a generator and a discriminator, characterized in that, The generator includes: A degradation removal sub-network for extracting features from an input image to obtain a first image ; The first image is a super-resolution and deblurred image; A priori estimation sub-network, which is used to obtain the style information in the prior knowledge according to the first image to obtain the style information in the prior knowledge ; the style information includes the color, shape and texture information of the iris image; A prior fusion sub-network for fusing a first image and style information to obtain prior fusion features; the prior fusion features are used to generate a clear restored image ; The prior fusion sub-network adopts a channel-aware attention mechanism; In the hourglass structure of the prior estimation sub-network, in the order from front to back, the output of the nth module among the last N residual channel attention modules is expressed as ; n is a natural number, and n ≤ N; In the aspect of fusing the first image and style information to obtain the prior fusion feature, the prior fusion sub-network is specifically used for: Perform N feature fusion operations to obtain prior fusion features; Among them, the nth feature fusion operation includes: Perform an affine transformation on using the convolution sum and mapping function, and convert into the weight coefficient in the channel attention mechanism ; Slice the feature map into spatial features with spatial information and spatial features ; According to the weight coefficient and spatial features obtain the style features; Concatenate the style feature and the spatial feature to obtain the fused feature corresponding to the nth feature fusion operation; In the first feature fusion operation, the feature map is the feature obtained after downsampling the first image and is the feature obtained after downsampling the first image In other secondary feature fusion operations, the feature map is the fusion feature corresponding to the previous feature fusion operation; The fusion feature obtained after the Nth feature fusion operation is used as the prior fusion feature.
2. The iris image restoration system based on prior guidance according to claim 1, characterized in that, The prior estimation sub-network includes a plurality of residual channel attention modules connected in a hourglass structure, a skip connection mechanism is adopted between the symmetric layers of the hourglass structure, and a convolutional layer is added between the symmetric layers; Any one of the symmetric layers includes: two residual channel attention modules symmetrically distributed on both sides of the center of the hourglass structure; The last N residual channel attention modules in the prior estimation sub-network are connected to the prior fusion sub-network, and the style information includes the outputs of the last N residual channel attention modules; N is a positive integer.
3. The iris image restoration system based on prior guidance according to claim 1, characterized in that, The degradation removal sub-network specifically includes a convolutional layer, three residual channel attention modules, and two convolutional layers connected in sequence.
4. The iris image restoration system based on prior guidance according to claim 2, wherein, The loss function of the generator is: Among them, , , , are the reconstruction loss, perceptual loss, style loss, and adversarial loss respectively, , , , are the trade-off parameters respectively; The reconstruction loss ; y is the real image, and ||||1 represents the L1 norm; The perceptual loss ; The perceptual loss is calculated by inputting the restored image and the real image y into the VGG-19 network, is the activation map of the i-th layer of the VGG-19 network; N i is the total number of pixels of the activation map of the i-th layer; The style loss ; is the Gram matrix constructed from the activation map of the i-th layer , where is the number of channels of the activation map of the i-th layer; The adversarial loss ; is the discriminator, representing the error between the real image y and the restored image , and E[ ] is the calculation of the expected value.