A seal deep learning recognition method based on adaptive image restoration
By employing an adaptive image restoration deep learning method for seals, and utilizing ST-HyperIQA and GG-YOLOv5s recognition networks, the problem of recognizing old and worn seal images was solved, achieving efficient recognition of diverse seal images.
Patent Information
- Application Number
- CN202410896667.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-05
AI Technical Summary
Existing deep learning-based seal recognition methods suffer from unsatisfactory recognition results when dealing with old and worn seal images, especially when handling interference such as blurred images, missing seal edges, missing characters inside the seal, and foreign objects obscuring the image, resulting in insufficient recognition accuracy.
An adaptive image inpainting deep learning method for seal recognition is adopted. Image quality is evaluated by ST-HyperIQA, and the multi-scale stitching capability of the network is optimized by combining the Swin-Transformer encoder. Selective inpainting is performed using the improved RDDM, and seal detection and recognition are performed by GG-YOLOv5s. A Group-GhostNetV2 bottleneck block and a feature fusion pyramid network are designed to enhance multi-scale feature extraction.
It improves the ability to recognize old and worn seal images, enhances the accuracy of image quality assessment and the precision of seal recognition, and adapts to the needs of seal image recognition under different conditions.
Smart Images

Figure CN118537267B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of target recognition based on image restoration, and in particular, relates to a seal deep learning recognition method based on adaptive image restoration. Background Art
[0002] As society develops, the number of paper seals and electronic seals is growing exponentially. The traditional manual seal recognition method is very costly. On this basis, a seal recognition method based on deep learning has been developed.
[0003] However, since seal recognition usually has a certain degree of interference problems, such as blurred images, missing seal edges, missing characters inside the seal, and occlusion by foreign objects, ordinary detection and recognition networks are not ideal for recognizing some old seals.
[0004] Regarding image restoration, Liu et al. (Jiawei Liu, Qiang Wang, Huijie Fan, YinongWang, Yandong Tang, and Liangqiong Qu. Residual denoising diffusion models. arXiv preprint arXiv:2308.13712, 2023.) mentioned that in real life, diffusion often occurs in a complex form involving multiple concurrent processes, such as the diffusion of multiple gases or the propagation of different types of waves or fields, which makes us think about whether the denoising-based diffusion model has limitations in focusing only on denoising. Therefore, this patent uses residual-guided forward diffusion and inverse generation of image restoration to selectively repair the seal.
[0005] Regarding image quality assessment, Su et al. (Shaolin Su, Qingsen Yan, Yu Zhu, ChengZhang, Xin Ge, Jinqiu Sun, Yanning Zhang, Blindly assess image quality in the wild guided by a self-adaptive hyper network, in: Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2020, pp. 3667–3676.) mentioned that since images captured in natural scenes contain different contents and different types of distortion, blind image quality assessment of real distorted images has always been a challenging problem. Therefore, this patent designs ST-HyperIQA (Swin-Transformer HyperIQA) to evaluate image quality, combining text prompts and semantic features to increase the real perception ability of the image quality assessment network.
[0006] Regarding the convolution strategy, Chen et al. (Yuming Chen, Xinbin Yuan, Ruiqi Wu, JiabaoWang, Qibin Hou, and Ming-Ming Cheng. YOLO-MS: rethinking multi-scalerepresentation learning for real-time object detection. arXiv preprint arXiv:2308.05480, 2023.) mentioned that multi-scale feature representation, as an important topic of modern target detectors, has a great impact on detection performance. Therefore, this patent designs Group-GhostNetV2 and feature fusion pyramid network to enhance the multi-scale feature extraction fusion capability and enhance the recognition network gradient flow.
[0007] In summary, although the seal image recognition method based on deep learning has made some progress, there are still some defects that need to be optimized. To this end, this patent proposes a seal deep learning recognition method based on adaptive image restoration to meet the diverse seal image recognition needs. Summary of the invention
[0008] The present invention aims to overcome the deficiencies in the above-mentioned prior art and designs a seal deep learning recognition method based on adaptive image restoration to improve the recognition ability of old and worn seal images.
[0009] In order to achieve the above invention, the present invention provides a seal deep learning recognition method based on adaptive image restoration, such as Figure 1 and Figure 2 As shown ( Figure 1 Flow chart of the embodiment of the present invention, Figure 2 , which is a structural diagram of an embodiment of the present invention, specifically comprises the following steps:
[0010] (1) Collecting seal image samples under various conditions
[0011] The collection includes seal image samples under various conditions, which means collecting seal image samples in different backgrounds, angles and distances, so that the seal image samples include pictures with different seal shapes, seal colors, foreign body occlusions, different scales and multi-angle rotations.
[0012] (2) Annotation and data augmentation of seal images
[0013] The seal image annotation and data augmentation described above refers to adjusting the size of the input seal image to a uniform size of 608 by using the OpenCV library. 2 The LabelImage annotation tool is used to annotate the minimum bounding rectangle of the seal to be detected, and the entire image is annotated with a text description of the scene. At the same time, Mosaic data augmentation is used to expand the data set, add Gaussian noise and salt and pepper noise to the image, and randomly change the saturation and brightness to simulate the diverse data in the real environment.
[0014] (3) Evaluate image quality using ST-HyperIQA
[0015] The image quality evaluation by ST-HyperIQA mentioned above refers to adding a Swin-Transformer encoder and optimizing the network's multi-scale splicing capabilities;
[0016] Furthermore, the addition of the Swin-Transformer encoder refers to adding a network that encodes text prompts using a Swin-Transformer encoder, taking into account that the semantic weights generated by HyperIQA through the content understanding hypernetwork are not sufficiently adaptive to the quality prediction target network parameters, and bidirectionally fusing the encoded information with the content understanding hypernetwork to correct the potential semantic understanding deviation;
[0017] Furthermore, the optimization network multi-scale splicing capability refers to adding EMA (Efficient Multi-Scale Attention) attention before the multi-scale feature splicing at the connection between the HyperIQA semantic feature extraction network and the quality prediction target network, and removing the global average pooling of the last layer of the HyperIQA semantic feature extraction network; then mapping the HyperIQA image quality evaluation score to a one-hot vector, and the evaluation score is greater than or equal to Mapping is 0, otherwise it is 1. is the learning factor.
[0018] (4) Selectively repair images by improving the loss function of RDDM (Residual Denoising Diffusion Models)
[0019] The selective restoration of the image by improving the RDDM loss function refers to the design of a color loss function based on the rich gradient information formed by the mainstream color and contour of the seal, taking into account the shortcomings of RDDM in restoring the seal characters in the image. , gradient loss and perceived loss Composite loss function , The specific formula of the loss function is as follows:
[0020] (1)
[0021] (2)
[0022] (3)
[0023] (4)
[0024] (5)
[0025] in, M is the number of pixels of the image, N is the number of feature layers, is the input image, For the output image, For images Gao Wei , width The pixel map value of the pixel at For red, blue and black in RGB colors, For non-red, blue and black in RGB colors, , For images , In the network The feature representation of the layer, , For images , Feature representation solved by HOG gradient, , , , , is the learning factor.
[0026] (5) Seal detection and recognition using GG-YOLOv5s
[0027] The seal detection and recognition by GG-YOLOv5s mentioned above refers to replacing the CSP (Cross Stage Partial) block with the optimized Group-GhostNetV2 bottleneck block and replacing the original neck network with the feature fusion pyramid network;
[0028] Furthermore, the use of the optimized Group-GhostNetV2 bottleneck block to replace the CSP block refers to the use of the optimized Group-GhostNetV2 bottleneck block to replace the CSP block for the YOLOv5s backbone network, that is, the strategy of group convolution is used to fully explore the multi-scale features, and channel shuffling is added after each depth-separable convolution layer, and the group convolution is spliced and the 1x1 convolution is used to adjust the number of channels. Finally, global average pooling is used at the end of the backbone instead of SPP (SpatialPyramid Pooling). The formula of group convolution is as follows:
[0029] (6)
[0030] in, Indicates the group number. is an integer greater than or equal to 1, represents the input features, represents the output features, is the convolution kernel size, is a convolutional block.
[0031] Furthermore, the use of a feature fusion pyramid network to replace the original neck network refers to the use of a feature fusion pyramid network to replace the original neck network of YOLOv5s, taking into account the large feature truncation problem of the original neck network, and performing different linear upsampling and linear downsampling on different feature maps to reduce the differences between different feature maps.
[0032] In terms of the training of GG-YOLOv5s and ST-HyperIQA, the Adam optimizer was used for training. After multiple rounds of training iterations converged, the image quality was evaluated and optional image restoration was used for seal detection and recognition. Among them, the parameters during training were set as follows: learning rate was 0.0003, learning rate decay factor was 0.8, deviation value was 0, linear layer weight was 0.02, deviation value was 0, Epochs was 500, and the training data size for each batch was 32.
[0033] Finally, after the image to be identified is scored by the image quality assessment network, the image is selectively repaired and transmitted to the seal recognition network, thereby realizing seal deep learning recognition based on adaptive image repair. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a flow chart of an embodiment of the present invention;
[0035] Figure 2 It is a structural diagram of an embodiment of the present invention;
[0036] Figure 3 : This is a diagram of (image quality assessment network) ST-HyperIQA according to an embodiment of the present invention;
[0037] Figure 4 This is a GG-YOLOv5s diagram of a (seal image recognition network) according to an embodiment of the present invention;
[0038] Figure 5 It is a module diagram of Group-GhostNetV2 according to an embodiment of the present invention; DETAILED DESCRIPTION
[0039] The technical route of the present invention is further described in detail below through specific embodiments and drawings.
[0040] like Figures 1 to 5 As shown, an embodiment of the seal deep learning recognition method based on adaptive image restoration of the present invention includes the following steps:
[0041] 1. Collect stamp image samples under various conditions
[0042] Collect seal image samples with different backgrounds, angles and distances, so that the seal image samples contain images with different seal shapes, seal colors, foreign body occlusions, different scales and multi-angle rotations.
[0043] 2. Labeling and data augmentation of seal images
[0044] Use OpenCV library to adjust the size of input stamp image to 608 2The LabelImage annotation tool is used to annotate the minimum circumscribed rectangle of the seal to be detected, and the entire image is annotated with a text description of the image scene; at the same time, Mosaic data augmentation is used to expand the data set, and 4 images are used as a group, and they are spliced in a random arrangement, random cropping and random scaling manner, that is, the RGB channel values in the image are arbitrarily changed, and a random value is added or subtracted to adjust the image brightness to generate 4 new images, and random occlusion processing is applied to the new images; then Gaussian noise and salt and pepper noise are added to the image to simulate the noise caused by paper aging and blurred shooting in a real environment.
[0045] 3. Evaluate image quality using ST-HyperIQA
[0046] Although the current seal image recognition method based on deep learning can recognize most seal images, some seal images need to be identified and repaired due to aging or blurred shooting. Therefore, this patent uses ST-HyperIQA to evaluate the image quality, aiming to identify images that need to be repaired; compared with the seal recognition network that directly detects and the method that does not identify and passes through the repair network and seal recognition network, this patent can achieve higher accuracy and speed.
[0047] Specifically, for ST-HyperIQA, considering that in practice, due to the different image contents, using the same rules to predict the quality of different images cannot fully cover their different structures, the following formula is used for modeling:
[0048] (7)
[0049] (8)
[0050] in, For input, That is, the network structure parameters change according to the semantics of the input. is the prediction score, For mapping, Extract the input features. is the network structure parameter.
[0051] ST-HyperIQA consists of four parts, namely, semantic feature extraction network, content understanding hypernetwork, quality prediction target network and text prompt embedding network.
[0052] Specifically, ResNet50 is used for semantic feature extraction, focusing on understanding the image content. The last layer of semantic features extracted by ResNet50 is input into the content understanding super network to generate weights, and the multi-scale features generated in multiple stages are simulated for local distortion perception and spliced through the EMA attention module before being input into the quality prediction target network. Among them, the EMA attention module reshapes part of the channel dimension into a batch dimension, fuses the output feature maps of two parallel sub-networks through a cross-space learning method, effectively establishes short-term and long-term dependencies, and removes the low-performance global average pooling in the original network.
[0053] At the same time, considering that only abstract semantic features can be extracted in HyperIQA, ST-HyperIQA is constructed by fusing Swin-Transformer encoder and HyperIQA, that is, the text prompt is fed into Swin-Transformer encoder for encoding to obtain text embedding, and the text embedding is used as input, and the input is used to update the text prompt embedding network, and the corrected semantic features are generated to generate more realistic weights and fed to the quality prediction target network, such as Figure 3 As shown. The quality prediction target network then simply maps the learned image content to a quality score, passes through four fully connected layers, accepts a multi-scale content feature vector as input, and propagates through the weights obtained by the content understanding hypernetwork to obtain the final image quality assessment score, with a value of 0 to 100. The image quality assessment score is then mapped to a one-hot vector,
[0054] That is, the evaluation score is greater than or equal to Mapping is 0, otherwise it is 1. is the learning factor.
[0055] 4. Selective image restoration via RDDM with improved loss function
[0056] RDDM is a double diffusion model based on residual transformation, which decouples the denoising diffusion process into residual diffusion and noise diffusion. Residual diffusion represents the directional diffusion from the target image to the degraded input image, and explicitly guides the reverse generation process of image restoration. Noise diffusion represents the random perturbation in the diffusion process. The model uses a general UNet network. The calculation formula for the degraded image with noise during the diffusion process is as follows:
[0057] (9)
[0058] in, is a degraded image with noise, is the target image, express , express , Residual information, For noise, Controls the speed at which the residuals diffuse, Controls the speed at which noise spreads.
[0059] Two models trained simultaneously during diffusion and Used to predict residuals and noise , the formula is as follows:
[0060] (10)
[0061] (11)
[0062] Among them, the hyperparameters and , which is used to control the prediction direction of the network for residual and noise.
[0063] At the same time, considering that there are deficiencies in repairing the seal characters in the image when RDDM is applied to seal image restoration, and combining the rich gradient information formed by the mainstream color and contour of the seal, a color loss , gradient loss and perceived loss Composite loss function , The specific formula of the loss function is as follows:
[0064] (12)
[0065] (13)
[0066] (14)
[0067] (15)
[0068] (16)
[0069] in, is the number of pixels in the image, is the number of feature layers, is the input image, For the output image, For images Gao Wei , width The pixel map value of the pixel at For red, blue and black in RGB colors, For non-red, blue and black in RGB colors, , For images , In the network The feature representation of the layer, For images , Feature representation solved by HOG gradient, , , , , is the learning factor.
[0070] Seal detection and recognition through GG-YOLOv5s
[0071] Taking into account that after the image passes through the quality assessment network and selectively passes through the image restoration network, the difficulty of image recognition transmitted to the seal recognition network is reduced. Therefore, this patent carries out a lightweight design of the seal recognition network, replaces the CSP block with the grouped convolution GhostNetV2 bottleneck block, and replaces the original neck network with a feature fusion pyramid network.
[0072] Specifically, Figure 4 , Figure 5 As shown in Figure 2, the Group-GhostNetV2 bottleneck block is used to replace the CSP block in YOLOv5s, that is, under the premise of retaining the depth-separable convolution and asymmetric convolution kernel of the original GhostNetV2 bottleneck block, the grouped convolution is used to divide the original features into Features , , add a skip connection to each non-first feature to balance the gradient flow, and add channel shuffle after each Ghost convolution block for information flow and dimensionality reduction between channels, then perform splicing and 1x1 convolution on the grouped convolution to adjust the number of channels. The formula for grouped convolution is as follows:
[0073] (17)
[0074] in, Indicates the group number. is an integer greater than or equal to 1, represents the input features, represents the output features, is the convolution kernel size, is a convolutional block.
[0075] At the same time, Group-GhostNetV2 also contains DFC attention, that is, decoupled fully connected attention, which aggregates pixels in the horizontal and vertical directions respectively, and saves tensor reshaping and transposition operations by sharing some transformation weights. The weight module includes reshaping, 3x3 convolution, fully connected layer and global average pooling. When inputting features, the Ghost module and DFC attention are two branches used to calculate the angle information and attention patches, and normalize and scale them through element-by-element multiplication and Sigmoid function, while retaining the long-range dependencies between pixels in different spatial positions, reducing unnecessary network parameters and accelerating the model's reasoning ability.
[0076] Then, the feature fusion pyramid network Figure 4 As shown, the multi-scale features extracted by Stage 2, Stage 3 and Stage 4 are fused, and different linear upsampling and linear downsampling are performed on different feature maps to reduce the differences between different feature maps.
[0077] Specifically, the feature fusion pyramid network scales the feature map to 1 / 2, 1 / 3, 1 / 4, 1 / 6, etc., and adds medium-scale feature maps to make the transition between features of different scales smoother and reduce scale truncation. The feature fusion pyramid network consists of multiple top-down and bottom-up synthesis fusion modules. Each synthesis fusion module has three optional linear scaling input size options, and outputs features of different sizes through pixel-by-pixel addition and 3x3 convolution. Multiple synthesis fusion modules jointly radiate and transmit to the detection head for seal recognition, thereby adapting to seal images of different sizes.
[0078] In terms of the training of GG-YOLOv5s and ST-HyperIQA, the Adam optimizer was used for training. After multiple rounds of training iterations converged, the image quality was evaluated and optional image restoration was used for seal detection and recognition. Among them, the parameters during training were set as follows: learning rate was 0.0003, learning rate decay factor was 0.8, deviation value was 0, linear layer weight was 0.02, deviation value was 0, Epochs was 500, and the training data size for each batch was 32.
[0079] Compared with the prior art, the present invention has the following beneficial effects:
[0080] (1) This patent designs the ST-HyperIQA module based on HyperIQA, improves the image quality assessment, encodes the text prompts by the Swin-Transformer encoder, and bidirectionally fuses the encoded information with the content understanding hypernetwork, corrects the potential semantic understanding bias of HyperIQA, enhances the adaptive ability of semantic feature understanding, and optimizes the accuracy of image quality assessment.
[0081] (2) Based on RDDM, this patent designs a loss function composed of gradient loss, color loss and perceptual loss. Taking into account the basic character features and color features of the seal, the HOG gradient feature representation, color loss and perceptual loss are used to specifically enhance the RDDM's ability to repair seal images.
[0082] (3) This patent designs the GG-YOLOv5s module based on YOLOv5s. By replacing the CSP block with the optimized Group-GhostNetV2 bottleneck block and replacing the original neck network with a feature fusion pyramid network, it fully mines multi-scale features, reduces the differences between different feature maps, adapts to seal images of different sizes, and improves recognition efficiency and accuracy.
[0083] (4) This patent provides an implementable seal deep learning recognition method based on adaptive image restoration, designs the ST-HyperIQA module, GG-YOLOv5s module and composite loss function, which greatly improves the seal recognition effect of the aged and blurred images to be restored, and provides a feasible method for seal deep learning recognition based on adaptive image restoration.
[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included in the scope of protection of the claims of the present invention.
Claims
1. A seal deep learning recognition method based on adaptive image restoration, It is characterized in that First, the image quality is evaluated by ST-HyperIQA, i.e. Swin-TransformerHyperIQA. Then, images with poor quality scores are selectively repaired by RDDM, i.e. ResidualDenoisingDiffusionModels, to achieve adaptive image repair. Finally, GG-YOLOv5s, i.e. Group-GhostNetV2, is constructed. YOLOv5s deep learning neural network is used to realize seal recognition, including the following steps: (1) Collecting seal image samples under various conditions: Collecting seal image samples under different backgrounds, angles and distances, so that the seal image samples contain pictures with different seal shapes, seal colors, foreign body occlusion, different scales and multi-angle rotations; (2) Annotating and data augmenting the seal image: Using the OpenCV library to adjust the size of the input original image to 608×608, and using the LabelImage annotation tool to annotate the minimum bounding rectangle of the seal to be detected, and annotating the entire image with a text description of the image scene, while using Mosaic data augmentation, adding Gaussian noise and salt and pepper noise to the image, and randomly changing the saturation and brightness to simulate the diverse data in the real environment; (3) Evaluating the image quality through ST-HyperIQA: Introducing the Swin-Transformer encoder to encode the text prompt, adding EMA, i.e. Efficient Multi-Scale Attention, before the multi-scale feature splicing at the connection between the HyperIQA semantic feature extraction network and the quality prediction target network; (4) Selectively repair the image by improving the RDDM loss function: Considering that the original RDDM loss function is insufficient in evaluating the restoration degree of the seal characters in the image, the rich gradient information formed by the mainstream color and contour of the seal is designed by combining the color loss L Color , gradient loss L Gradient and the perceptual loss L Perceptual The composite loss function L New , L New The specific formula of the loss function is as follows: L New =b 1 L Gradient +b 2 L Color +b 3 L Perceptual (1)L Gradient =log 2 (|g(x)-g(y)|) (2) Where M is the number of pixels in the image, N is the number of feature layers, x is the input image, y is the output image, and f(x i,j is the pixel mapping value of the pixel at the position of image x with height i and width j, t 1 is the red, blue and black tuple in RGB color, t 2 is a non-red, blue, and black tuple in RGB color, F(x i '), F(y i ') is the feature representation of image x and y in the i-th layer of the network, g(x) and g(y) are the feature representations of image x and y solved by HOG gradient, β 1 , β 2 , β 3 , α, ε are learning factors; (5) Seal detection and recognition through GG-YOLOv5s: the optimized Group-GhostNetV2 bottleneck block is used to replace the CSP, that is, the Cross StagePartial block, and the feature fusion pyramid network is used to replace the original neck network.
2. According to the seal deep learning recognition method based on adaptive image restoration according to claim 1, It is characterized in that The image quality evaluation by ST-HyperIQA refers to adding a network that encodes text prompts by a Swin-Transformer encoder, and bidirectionally fusing the encoded information with the content understanding hypernetwork to correct potential semantic understanding deviations, taking into account that the semantic weights generated by HyperIQA through the content understanding hypernetwork are not sufficiently adaptive to the quality prediction target network parameters; at the same time, EMA attention is added before the multi-scale feature splicing at the connection between the HyperIQA semantic feature extraction network and the quality prediction target network, and the global average pooling of the last layer of the HyperIQA semantic feature extraction network is removed; then the image quality evaluation score of HyperIQA is mapped to a One-hot vector, and the evaluation score greater than or equal to θ is mapped to 0, otherwise it is mapped to 1, and θ is a learning factor.
3. According to the seal deep learning recognition method based on adaptive image restoration described in claim 1, It is characterized in that The seal detection and recognition is performed by GG-YOLOv5s. The GG-YOLOv5s refers to a network obtained by improving the backbone network of YOLOv5s and the neck network of YOLOv5s. The improved backbone network of YOLOv5s refers to replacing the CSP block with the optimized Group-GhostNetV2 bottleneck block for the backbone network of YOLOv5s, that is, using the strategy of group convolution to fully explore multi-scale features, adding channel shuffling after each depth-separable convolution layer, and splicing the group convolution and adjusting the number of channels by 1x1 convolution. Finally, global average pooling is used at the end of the backbone to replace SPP, that is, SpatialPyramid Pooling. The formula of group convolution is as follows: Among them, i represents the grouping number, i is an integer greater than or equal to 1, X represents the input feature, Y represents the output feature, k is the convolution kernel size, and C is the convolution block; the improved neck network of YOLOv5s refers to the neck network of YOLOv5s, which uses a feature fusion pyramid network to replace the original neck network, and performs different linear upsampling and linear downsampling on different feature maps to reduce the differences between different feature maps.
