Seal character detection and recognition method and device

Abstract

This invention relates to a method for detecting and recognizing seal characters, comprising initializing the detection and recognition of a seal, acquiring a seal image using an image acquisition module, establishing a seal character detection model, detecting characters in the acquired seal image, performing horizontal correction on the detection results using polar coordinate transformation, establishing a seal character recognition model, recognizing characters in the corrected seal character image, and outputting the seal character recognition result; realizing the detection and recognition of seal characters through the seal character detection model and the seal character recognition model, and displaying the seal image through a display module connected to a computer, accurately recognizing seals with blurriness, variable aspect ratios, and overlap with background text, thereby improving the efficiency of verification and classification.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for detecting and recognizing seal characters. Background Technology

[0002] Seals, as legally significant credentials, play an important role in various sectors of Chinese society. With the continuous development of modern business activities, enterprises often involve a large amount of contract signing and archiving work during business operations. If manual verification of the seal content in document images is still used, it will not only consume a lot of manpower but also be inefficient. Therefore, a deep learning-based seal character detection and recognition method can accurately extract seal characters, thereby enabling computers to replace manual verification, improve the efficiency of contract verification and classification, reduce business risks in the process of signing financial and tax contracts, and make business connections more efficient and convenient.

[0003] Currently, there are two technical solutions for seal detection and recognition: one is based on traditional methods for seal character detection and recognition, but the performance of traditional methods largely depends on the effectiveness of manual features, resulting in the inability to accurately detect and recognize seal characters in complex backgrounds; the other solution uses deep learning-based methods for seal character detection and recognition, which has high accuracy and wide application range, but due to the characteristics of seals such as fuzziness, varying aspect ratios, and overlap with background text, the seal character detection and recognition effect cannot reach the best.

[0004] Chinese patent CN 115546811 A discloses a method, apparatus, device, and storage medium for recognizing seals. The method includes: acquiring an image to be recognized; extracting a seal image from the image to be recognized according to a preset seal detection model; matching at least one preset seal shape with the seal image to determine the text direction of target characters in the seal image; converting the text direction of the target characters to a preset direction; classifying and recognizing the target characters converted to the preset direction to obtain characters of at least one character type; and outputting the characters of the at least one character type. This method can effectively recognize characters in seals and improve the recognition effect of characters in seals. However, this patent cannot solve the problem of seal recognition with characteristics such as blurriness, variable aspect ratio, and overlap with background text. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting and recognizing seal characters. It achieves the detection and recognition of seal characters through a seal character detection model and a seal character recognition model, and displays the seal image through a display module connected to a computer. It can accurately recognize seals that are blurry, have varying aspect ratios, and overlap with background text, thereby improving the efficiency of verification and classification.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] See Figure 1 A method for detecting and recognizing seal characters, comprising the following steps:

[0008] Step 1: Initialize seal detection and recognition;

[0009] Step 2: Use the image acquisition module to acquire the image of the seal;

[0010] Step 3: Establish a seal character detection model by using a feature extraction backbone network, a receptive field enhancement module, a feature fusion network, a color feature extraction module, and an output layer to detect characters in the collected seal images;

[0011] Step 4: Perform horizontal correction on the detection results using polar coordinate transformation;

[0012] Step 5: Establish a seal character recognition model using Patch Embedding, CAFM, Transformer Encoder, CNN neural network, and Transformer Decoder to perform character recognition on the corrected seal character image;

[0013] Step 6: Output the seal character recognition results.

[0014] Furthermore, the initialization of the detection and recognition of seal characters in step 1 includes the following steps:

[0015] Step 1-1: Initialize the acquired images;

[0016] Step 1-2: Initialize image parameters;

[0017] Steps 1-3: Load the parameters of the seal character detection model. Read and input the weight parameters and hyperparameters of the seal character detection model from the database, and load the parameters into the seal character detection model.

[0018] Steps 1-4: Load the seal character recognition model parameters. Read and input the weight parameters and hyperparameters of the seal character recognition model from the database, and load the parameters into the seal character recognition model.

[0019] Furthermore, in step 3, the feature extraction backbone network extracts features from the character image of the seal to be inspected, enhances the receptive field of the extracted feature map through the receptive field enhancement module, and the feature fusion network fuses the enhanced feature map layer by layer to obtain the fused feature map of the character image of the seal to be inspected. The color feature extraction module extracts color features from the character image of the seal to be inspected to obtain the color feature map of the seal character image. The fused feature map and the color feature map are fused to determine the character target in the character image of the seal to be inspected, and the fused feature map is output through the output layer.

[0020] Furthermore, in step 3, the receptive field enhancement module uses multi-scale convolutional kernels to form a multi-branch structure.

[0021] Furthermore, step 4 involves using polar coordinate transformation to perform horizontal correction on the detection results, including the following steps:

[0022] Step 4-1: Read the seal character image and character region coordinates;

[0023] Step 4-2: Extract the image of the seal characters based on the coordinates of the character region;

[0024] Step 4-3: Perform horizontal correction on the extracted seal character image using polar coordinate transformation;

[0025] Step 4-4: Output the corrected image of the seal characters.

[0026] Furthermore, in step 5, the character recognition encoder Transformer Encoder uses the self-attention model Vision Transformer to obtain the global feature map of the seal character image, the CNN neural network extracts features from the seal character image to be recognized to obtain a shallow feature map, and the feature fusion module CAFM is used to fuse the global feature map and the shallow feature map to obtain a fused feature map. The character recognition decoder Transformer Decoder generates a text sequence and outputs the recognition result through a fully connected layer.

[0027] A seal character detection and recognition device includes a data acquisition module, a detection module, a recognition module, a display module, a database module, and a computer. The computer contains the detection module, the recognition module, and the database module. The data acquisition module is connected to the computer, and the computer is connected to the display module.

[0028] Acquisition module: Includes a camera and a ring light source, used for real-time acquisition of seal character images;

[0029] Detection module: Includes a seal character detection model, used to detect character regions in seal images;

[0030] Recognition module: Includes a seal character recognition model, used to recognize the text in a seal image;

[0031] Display module: Used to display real-time acquisition status, detection result image, recognition result image, and recognized text;

[0032] Database module: Used to store data and parameters for the seal character detection and recognition method;

[0033] Computer equipment: used to drive the seal character image acquisition and input device and to run the seal character detection and recognition method program.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] 1) By using the receptive field enhancement module in the seal character detection model, the detection accuracy of characters with different aspect ratios is effectively improved. The color feature extraction module is used to extract red feature information in the seal image, which improves the detection accuracy of blurry and incomplete seal character images.

[0036] 2) Establish a seal character recognition model. Obtain global feature maps through the self-attention model Vision Transformer and local feature maps through the CNN neural network. Input these features into the parallel feature fusion module CAFM for fusion to obtain robust features and reduce the impact of arbitrary-shaped text and blurred or incomplete characters on the model's recognition accuracy.

[0037] 3) It effectively improves the accuracy of seal character detection and recognition, the efficiency of seal character review, and can be well applied to actual work scenarios. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0039] Figure 2 This is a schematic diagram of the process for initializing seal character detection and recognition as described in this invention.

[0040] Figure 3 This is a structural block diagram of the seal character detection model described in this invention.

[0041] Figure 4 This is a block diagram of the receptive field enhancement module described in this invention.

[0042] Figure 5 This is a structural block diagram of the color feature extraction module described in this invention.

[0043] Figure 6 This is a schematic diagram of the seal correction process described in this invention.

[0044] Figure 7 This is a structural block diagram of the seal character recognition model described in this invention.

[0045] Figure 8 This is a structural block diagram of the seal character detection and recognition device described in this invention.

[0046] Figure 9 This is a schematic diagram of the hardware structure of the seal character detection and recognition device described in this invention.

[0047] In the image: 1. Camera 2. Ring light source 3. Computer 4. Monitor Detailed Implementation

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0049] See Figure 1 This is a schematic diagram of the method flow of the present invention. The present invention provides a method for detecting and recognizing seal characters, comprising the following steps:

[0050] Step 1: Initialize seal detection and recognition;

[0051] Step 2: Use the image acquisition module to acquire the image of the seal;

[0052] Step 3: Establish a seal character detection model by using a feature extraction backbone network, a receptive field enhancement module, a feature fusion network, a color feature extraction module, and an output layer to detect characters in the collected seal images;

[0053] Step 4: Perform horizontal correction on the detection results using polar coordinate transformation;

[0054] Step 5: Establish a seal character recognition model using the Patch Embedding layer, the CAFM feature fusion module, the Transformer Encoder, the CNN neural network, and the Transformer Decoder to perform character recognition on the corrected seal character image;

[0055] Step 6: Output the seal character recognition results.

[0056] See Figure 2 Step 1, which initializes the detection and recognition of seal characters, includes the following steps:

[0057] Step 1-1: Initialize the acquired images;

[0058] Step 1-2: Initialize image parameters;

[0059] Steps 1-3: Load the parameters of the seal character detection model. Read and input the weight parameters and hyperparameters of the seal character detection model from the database, and load the parameters into the seal character detection model.

[0060] Steps 1-4: Load the parameters of the seal character recognition model. Read and input the weight parameters and hyperparameters of the seal character recognition model from the database, and load the parameters into the seal character recognition model. The weight parameters and hyperparameters of the seal character detection model and the character recognition model are obtained by offline training of text detection and text recognition respectively through the seal character image dataset, and then stored in the database.

[0061] Step 2: Submit the document containing the seal content to the seal image acquisition area for seal image acquisition.

[0062] Step 3: Use a seal character detection model to detect the characters in the seal;

[0063] See Figure 3 The seal character detection model is based on the TextSnake model and includes a feature extraction backbone network, a receptive field enhancement module, a feature fusion network, a color feature extraction module, and an output layer. The feature extraction backbone network extracts features from the seal character image to be detected, obtaining a first, second, third, and fourth predicted feature map. Next, the receptive field enhancement module enhances the receptive fields of these four maps. Then, the feature fusion network fuses the enhanced feature maps layer by layer to obtain a fused feature map of the seal character image to be detected. Simultaneously, the color feature extraction module extracts color features from the seal character image to obtain a color feature map. Finally, the fused feature map and the color feature map of the seal character image to be detected are fused to determine the target character in the image.

[0064] In the implementation of the seal character detection model, ResNet50 is used as the feature extraction backbone network. The ResNet50 network contains 4 convolutional groups and 1 convolutional layer. Each convolutional group consists of residual modules composed of 2 1×1 and 1 3×3 convolutional layers. The acquired seal image is processed by the feature extraction backbone network ResNet50 to obtain the first, second, third, and fourth predicted feature maps of the seal character image. The structure of the feature extraction network ResNet50 is shown in Table 1.

[0065] Table 1

[0066]

[0067] The enhanced feature maps are fused layer by layer through the feature fusion network to obtain a fused feature map of the character image of the seal to be inspected. The feature fusion network adopts a top-down approach to perform multi-scale feature fusion. In each fusion stage, feature map h1 is upsampled and magnified by a factor of two, and then tensor concatenated with the current feature map m2. 3×3 convolution is used for feature fusion, and 1×1 convolution is used to increase the nonlinearity of the network and reduce the number of channels of the feature map to obtain feature map h2. The above operations are performed on feature maps m2, m3, and m4 respectively to obtain fused feature map h4. Feature map h4 is upsampled and magnified by a factor of two to obtain the final fused feature map.

[0068] See Figure 4 The receptive field enhancement module in the seal character detection model uses multi-scale convolutional kernels to form a multi-branch structure. The input feature map is subjected to four 1×1 convolutional kernels to change the number of channels in the feature map. The four branches are subjected to 1×3, 3×1, 1×5, and 5×1 convolutions to extract features at multiple scales to enrich spatial features and increase feature diversity. The first, second, and third branches are divided into two branches after 1×1 convolution. One branch is input to the next set of convolutions and is deeply fused with the lower convolution to retain the texture information of the upper layer features. The other branch directly outputs the global feature information at that scale. The output features of each branch are concatenated by tensors and then fused at multiple scales through 1×1 convolution to enrich the shallow texture information and deep semantic information of the features. Finally, the multi-dimensional fused features are concatenated with the original features to generate the final feature map, so that the output features retain more original feature information.

[0069] See Figure 5 The color feature extraction module in the seal character detection model uses a 3×3 convolution kernel to extract features from the input seal character image, and a channel attention module to extract red features. The channel attention module is then used to extract the red features of this feature map in depth. Finally, the obtained color features are concatenated with the original image and fused by a 1×1 convolution to generate the final color feature map.

[0070] In the color feature extraction module, the original seal character image X has a length, width, and number of channels H, W, and C, respectively. X is first processed by a convolutional module with a kernel size of 3×3 and a stride of 1 to obtain F. The feature F is then processed by the channel attention mechanism CAM to obtain the channel attention weights M. c Then multiply it with the feature F to obtain the output feature F', and obtain the channel attention weight M' of feature F' through CAM. c Then, multiply the result by the feature F' and sum them together. Finally, use a 1×1 convolution to reduce the feature dimension to obtain the output feature F'. The calculation formula is as follows:

[0071]

[0072]

[0073]

[0074]

[0075] In the formula, σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, Maxpool represents max pooling, Avgpool represents average pooling, ReLU represents the ReLU activation function, and Conv represents 1×1 convolution.

[0076] Finally, after element-wise addition of features F” and X, a 1×1 convolution is used to increase the channel features, thereby obtaining the final output feature Y, the calculation formula of which is shown below:

[0077] Y = Conv(X + F”) (5)

[0078] In the formula, Conv represents 1×1 convolution.

[0079] Step 4: Correct the detection results using polar coordinate transformation. The specific steps are as follows:

[0080] See Figure 6 Step 4-1: Read the seal character image and character region coordinates;

[0081] Step 4-2: Extract the image of the seal characters based on the coordinates of the character region;

[0082] Step 4-3: Perform horizontal correction on the extracted seal character image using polar coordinate transformation;

[0083] Step 4-4: Output the corrected image of the seal characters.

[0084] Step 5: Input the corrected seal character image into the seal character recognition model for character recognition;

[0085] See Figure 7The seal character recognition model includes a patch embedding layer, a feature fusion module (CAFM), a character recognition encoder (Transformer Encoder), a CNN neural network, and a character recognition decoder (Transformer Decoder). The character recognition encoder (Transformer Encoder) uses a self-attention model (Vision Transformer) to obtain a global feature map of the seal character image. The CNN neural network extracts features from the seal character image to be recognized to obtain a shallow feature map. The feature fusion module (CAFM) fuses the global feature map and the shallow feature map to obtain a fused feature map. The character recognition decoder (Transformer Decoder) generates a text sequence and outputs the recognition result through a fully connected layer.

[0086] Furthermore, the self-attention model, Vision Transformer, consists of two parts: a patch embedding layer and a character recognition encoder, Transformer Encoder. The input image undergoes a linear transformation through the patch embedding layer to obtain a 2D feature vector. This 2D feature vector is then processed by the character recognition encoder, Transformer Encoder, to obtain the global features of the seal character image. The specific process is as follows:

[0087] The input image is divided into N by the Patch Embedding layer. 2 The image is a block image, and N is obtained through a linear transformation. 2 Block global semantic features, where global semantic features T and T cls The learned tags are concatenated together, and the concatenation result is combined with the positional encoding information PE. pos Add them together to get the final input T. in :

[0088]

[0089] The character recognition encoder (Tranformer Encoder) comprises a multi-head self-attention (MSA) mechanism and a multi-layer perceptron (MLP). The MSA mechanism consists of multiple superimposed self-attention layers. in Normalization is performed through a Layer Norm (LN) layer, and the input is fed into a multi-head self-attention layer to obtain the output feature T of MSA. MSA :

[0090] T MSA =Tin +MSA(LN(T in (7)

[0091] Among them, T MSA The output features of the multi-head self-attention mechanism;

[0092] Then, the output feature T of the MSA layer MSA After LN normalization, the input is fed into the MLP layer to obtain the output of the coding layer:

[0093] T out =T MSA +MLP(LN(T attn (8)

[0094] Therefore, for the next coding layer, Tout becomes T. in Input, and repeat this process K times to obtain the final global feature map;

[0095] Furthermore, in the CNN neural network, each convolutional layer is followed by a 2×2 max-pooling layer, and there are a total of two 3×3 convolutional layers. The input data is processed twice by the convolutional layer and the pooling layer to obtain the local features of the seal character image.

[0096] Furthermore, the feature fusion module CAFM consists of two self-attention modules. First, the two self-attention modules fuse the global features extracted by the self-attention model Vision Transformer with the local features extracted by the shallow CNN neural network, resulting in global features fused with local features and local features fused with global features. The global and local features are then added to their corresponding original features to retain more original feature information. Finally, the global and local features are concatenated through a tensor concatenation operation to obtain the final output features.

[0097] Furthermore, the Transformer Decoder structure is similar to the Transformer Encoder structure, but it contains two sets of multi-head self-attention layers. The input character labels are processed by the Character Embedding layer to obtain the input vector. The obtained input vector is subjected to residual connection and normalization operations by Masked Multi-head Self Attention (MMSA) to obtain the output features of the MMSA layer. The output features and the output features of the encoder are passed to the MSA layer, and then through the MLP layer to obtain the output of the Transformer Decoder layer. In the implementation of the seal character recognition model, the output of the Transformer Decoder layer is linearly transformed and subjected to the softmax function to obtain the final seal character recognition result.

[0098] Step 6: Output the seal character recognition results, obtain the seal detection and recognition model processing results, and output and display them.

[0099] See Figure 8 A device for detecting and recognizing seal characters includes a data acquisition module, a detection module, a recognition module, a display module, a database module, and a computer. The computer contains the detection module, the recognition module, and the database module. The data acquisition module is connected to the computer, and the computer is connected to the display module.

[0100] See Figure 9 The display screen is a 24-inch LCD screen, the camera is a Vision Intelligent Manufacturing MV-HS2000GC, the camera is connected to the computer via an RJ45 network port, and the display screen is connected to the computer via an HDMI interface.

[0101] The detection module consists of a Visionox MV-2000GC camera and a Visionox AFT-RL12068W ring light source. It is used to acquire seal character images in real time, train the seal character detection model offline, obtain the optimal weight parameters and hyperparameters, and determine the current position of the seal through the character detection model.

[0102] Recognition Module: Includes a seal character recognition model, used to recognize text in seal images. The seal character recognition model is trained offline to obtain the optimal weight parameters and hyperparameters. The weight parameters and hyperparameters of the character recognition model are loaded, and the seal character image is recognized through the character recognition model to output the character recognition results.

[0103] Display module: It adopts a 24-inch LCD screen to display the real-time acquisition status, detection result image, recognition result image and recognized text, display the layout of the operation window, provide the operator with a visual operation area, display the real-time video information of the camera, display the seal character image detected by the seal character detection model, and display the seal character recognized by the seal character recognition model.

[0104] Database module: Solid-state storage hard disk is used to store the data and parameters of the seal character detection and recognition method, including the layout data of the operation window, the parameters of the seal character detection model, the parameters of the seal character recognition model, and the recognition results displayed in the operation window display area;

[0105] Computer: Used to drive the seal character image acquisition and input device and run the seal character detection and recognition method program. It is connected to the Vision Intelligent Manufacturing MV-HS2000GC camera and display screen, drives the Vision Intelligent Manufacturing MV-HS2000GC and display screen, runs the acquisition module and display module, runs the database module, saves the program, data and parameters of the character detection and recognition model through solid-state storage hard disk, runs the character detection model and character recognition model, completes the character detection and recognition function of seal character image, and provides GPU computing power for the model of character detection module and character recognition module.

[0106] The above embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the above embodiments. Unless otherwise specified, the methods used in the above embodiments are conventional methods.

Claims

1. A method for detecting and recognizing seal characters, characterized in that, Includes the following steps: Step 1: Initialize seal detection and recognition; Step 2: Use the image acquisition module to acquire the image of the seal; Step 3: Establish a seal character detection model by using a feature extraction backbone network, a receptive field enhancement module, a feature fusion network, a color feature extraction module, and an output layer to detect characters in the collected seal images; The feature extraction backbone network extracts features from the character image of the seal to be inspected. The receptive field enhancement module enhances the receptive field of the extracted feature map. The feature fusion network fuses the enhanced feature map layer by layer to obtain the fused feature map of the character image of the seal to be inspected. The color feature extraction module extracts color features from the character image of the seal to be inspected to obtain the color feature map of the seal character image. The fused feature map and the color feature map are fused to determine the character target in the character image of the seal to be inspected. The fused feature map is output through the output layer. Step 4: Perform horizontal correction on the detection results using polar coordinate transformation; Step 5: Establish a seal character recognition model using Patch Embedding, CAFM, Transformer Encoder, CNN neural network, and Transformer Decoder to perform character recognition on the corrected seal character image; The character recognition encoder (Transformer Encoder) uses the self-attention model (Vision Transformer) to obtain the global feature map of the seal character image. The CNN neural network extracts features from the seal character image to be recognized to obtain a shallow feature map. The feature fusion module (CAFM) is used to fuse the global feature map and the shallow feature map to obtain a fused feature map. The character recognition decoder (Transformer Decoder) generates a text sequence and outputs the recognition result through a fully connected layer. Step 6: Output the seal character recognition results.

2. The method for detecting and recognizing seal characters according to claim 1, characterized in that, The initialization of seal detection and recognition in step 1 includes the following steps: Step 1-1: Initialize the acquired images; Step 1-2: Initialize image parameters; Steps 1-3: Load the parameters of the seal character detection model. Read and input the weight parameters and hyperparameters of the seal character detection model from the database, and load the parameters into the seal character detection model. Steps 1-4: Load the seal character recognition model parameters. Read and input the weight parameters and hyperparameters of the seal character recognition model from the database, and load the parameters into the seal character recognition model.

3. The method for detecting and recognizing seal characters according to claim 1, characterized in that, In step 3, the receptive field enhancement module uses multi-scale convolutional kernels to form a multi-branch structure.

4. The method for detecting and recognizing seal characters according to claim 1, characterized in that, Step 4 involves using polar coordinate transformation to perform horizontal correction on the detection results, including the following steps: Step 4-1: Read the seal character image and character region coordinates; Step 4-2: Extract the image of the seal characters based on the coordinates of the character region; Step 4-3: Perform horizontal correction on the extracted seal character image using polar coordinate transformation; Step 4-4: Output the corrected image of the seal characters.

5. The apparatus for a seal character detection and recognition method according to any one of claims 1-4, characterized in that, It includes a data acquisition module, a detection module, an identification module, a display module, a database module, and a computer. The computer is equipped with the detection module, the identification module, and the database module. The data acquisition module is connected to the computer, and the computer is connected to the display module. Acquisition module: Includes a camera and a ring light source, used for real-time acquisition of seal character images; Detection module: Includes a seal character detection model, used to detect character regions in seal images; Recognition module: Includes a seal character recognition model, used to recognize the text in a seal image; Display module: Used to display real-time acquisition status, detection result image, recognition result image, and recognized text; Database module: Used to store data and parameters for the seal character detection and recognition method; Computer equipment: used to drive the seal character image acquisition and input device and to run the seal character detection and recognition method program.

Images