Seal identification method and device, equipment, storage medium and product
By determining the coordinates and categories of seals, performing image cutting and angle correction, and using appropriate text recognition models to identify seal text, the problem of low seal recognition accuracy in the prior art is solved, especially the recognition effect of curved text is significantly improved.
Patent Information
- Application Number
- CN202410996494.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, the seal recognition accuracy is low, especially the recognition effect of different types of text is poor, and the curved text is prone to text fracture and cumulative error problems during straightening.
By obtaining the seal image to be identified, determining the seal coordinate information and category, cutting the image, angle correction is performed according to the text type, and using the corresponding pre-trained model for text recognition, directly inputting the curved text for recognition, avoiding the straightening process.
It improves the accuracy and versatility of seal recognition, reduces the impact of text orientation on recognition, avoids text breakage and cumulative errors, and enhances the accuracy of identification of curved text.
Smart Images

Figure CN119068235A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to a seal recognition method, device, equipment, storage medium and product. Background Art
[0002] As a symbol and evidence with legal significance in a company's business behavior, seals are widely used in business scenarios such as contract comparison, risk assessment, and document authentication. For the review of contracts, bills and other documents in government and enterprise matters, the traditional method is mainly completed through manual review. The review content mainly involves checking whether the contract seal is used in the contract, whether the seal usage specifications are correct; whether the main text of the seal is consistent with the company name in the contract; whether the contract content is consistent with the approval, etc. With the development of artificial intelligence, seal text recognition technology can automatically identify the text information on the company seal, thereby providing great convenience in document archiving, digital archive management, and contract approval.
[0003] In recent years, deep learning technology, especially models such as convolutional neural network (CNN) and recurrent neural network (RNN), has made great progress in seal detection and recognition. These models can improve the accuracy of seal recognition to a certain extent by learning features from raw image data. However, due to the wide variety of seals, the common seals currently include round seals, oval seals, square seals, diamond seals and triangular seals, etc. The text types contained in different types of seals are different, including horizontal text, inclined text, vertical text and curved text, etc. In the prior art, the same model is used for recognition of different types of text. Due to the structural differences between texts, the same model may be difficult to adapt to the characteristics of all text types, resulting in low recognition accuracy for different types of text. In addition, for curved text in the seal, the prior art usually flattens the curved text into horizontal text before performing text recognition. In the straightening process, text breaks and accumulated errors are prone to occur, resulting in low recognition accuracy for curved text in the seal. In addition, how to improve the accuracy of seal recognition has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of the above problems in the prior art, the purpose of the embodiments of this specification is to provide a seal recognition method, device, equipment, storage medium and product to solve the problem of low seal recognition accuracy in the prior art.
[0005] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:
[0006] On the one hand, an embodiment of this specification provides a seal recognition method, the method comprising:
[0007] Obtaining a seal image to be identified;
[0008] Determining the seal coordinate information and seal category in the seal image to be identified;
[0009] Cutting the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified;
[0010] Determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category;
[0011] Performing angle correction on the initial seal text area according to the text type to obtain a first seal text area;
[0012] A pre-trained text recognition model corresponding to the text type is obtained, and the first seal text area is recognized using the text recognition model to obtain a first seal text recognition result.
[0013] Preferably, the determining the seal coordinate information in the seal image to be identified includes:
[0014] Inputting the seal image to be identified into a pre-trained seal detection model to obtain the coordinates of the center point of the rectangular area where the seal is located;
[0015] The seal coordinate information is determined according to the center point coordinates and the length and width values of the rectangular area.
[0016] Preferably, the step of determining the seal category in the seal image to be identified includes:
[0017] The seal category in the seal image to be identified is identified based on a pre-trained seal classification model, wherein the seal classification model is trained using historically collected seal images and seal category labels thereof.
[0018] Preferably, the determining of the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category includes:
[0019] Inputting the seal slice image to be identified into a pre-trained text detection model to obtain an initial seal text area;
[0020] Calculating the curvature of the initial seal text area;
[0021] If the curvature is greater than a preset threshold, and the seal type corresponding to the initial seal text area is circular or elliptical, determining that the text type is curved text;
[0022] If the curvature is less than a preset threshold and the seal type corresponding to the initial seal text area is non-circular or non-elliptical, the text type is determined to be non-curved text, wherein the non-curved text includes horizontal text, vertical text and inclined text.
[0023] Preferably, the step of performing angle correction on the initial seal text area according to the text type comprises:
[0024] If the text type is curved text, the PCA algorithm is used to perform angle correction on the initial seal text area;
[0025] If the text type is non-curved text, the initial seal text area is angle-corrected using a minimum circumscribed polygon.
[0026] Preferably, the step of using the PCA algorithm to perform angle correction on the initial seal text area includes:
[0027] Acquire the coordinate information of the initial seal text area, and extract the seal text image according to the coordinate information;
[0028] Performing a debinarization process on the seal text image to obtain a debinarized seal text image;
[0029] Obtaining all non-zero pixel coordinates in the debinarized seal text image;
[0030] Calculating the eigenvalues and eigenvectors of the non-zero pixel coordinates, and selecting the eigenvector with the largest eigenvalue as the principal component;
[0031] Calculating the angle between the principal component and the horizontal coordinate axis, and using the angle as the text rotation angle of the seal text image;
[0032] Angle correction is performed on the initial seal text area according to the text rotation angle.
[0033] Preferably, the text recognition model of the curved text includes a preprocessing module, a mapping module, an image transformer and a text transformer;
[0034] The step of using the text recognition model to recognize the first seal text area to obtain a first seal text recognition result includes:
[0035] Inputting the first seal text area into the preprocessing module to obtain a minimum bounding rectangle and a number of text blocks uniformly divided by the minimum bounding rectangle;
[0036] Inputting the text block into the mapping module for linear mapping and position embedding to obtain a text block embedding vector;
[0037] Inputting the text block embedding vector into the image Transformer to extract features and obtain feature representation;
[0038] The feature representation is input into the text transformer for feature decoding to obtain a first seal text recognition result.
[0039] Preferably, the method further comprises:
[0040] Matching the first seal text recognition result with the fixed fields in the pre-built fixed field library to obtain the corresponding matching confidence;
[0041] If the matching confidence is greater than a preset threshold, the first seal text recognition result is corrected according to the fixed field, and the corrected first seal text recognition result is determined as the final seal text recognition result.
[0042] Preferably, the method further comprises:
[0043] Vertically flipping the first seal text area to obtain a second seal text area;
[0044] Inputting the second seal text area into the corresponding pre-trained text recognition model for text recognition to obtain a second seal text recognition result;
[0045] The confidence levels of the first seal text recognition result and the second seal text recognition result are calculated, and the seal text recognition result with a higher confidence level is determined as the final seal text recognition result.
[0046] On the other hand, an embodiment of the present specification provides a seal recognition device, the device comprising:
[0047] An acquisition module, used for acquiring the seal image to be identified;
[0048] A seal positioning and classification module, used to determine the seal coordinate information and seal category in the seal image to be identified;
[0049] A seal slicing module, used for cutting the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified;
[0050] A text area determination and classification module, used to determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category;
[0051] A text angle correction module, used for performing angle correction on the initial seal text area according to the text type to obtain a first seal text area;
[0052] The text recognition module is used to obtain a pre-trained text recognition model corresponding to the text type, and use the text recognition model to recognize the first seal text area to obtain a first seal text recognition result.
[0053] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.
[0054] On the other hand, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.
[0055] On the other hand, the embodiments of this specification further provide a computer program product, which, when executed by a processor of a computer device, executes instructions of any one of the above methods.
[0056] One or more technical solutions provided by some embodiments of this specification have at least the following technical effects:
[0057] The embodiment of this specification obtains a slice image containing only the seal through the seal coordinate information of the seal image to be identified, and determines the seal text area and the corresponding text type according to the seal category, and adopts different correction methods for different text types to perform angle correction, thereby reducing the influence of the text direction on the seal text recognition. Furthermore, considering the structural differences between different types of texts, the embodiment of this specification selects different text recognition models for different text types for text recognition, which improves the versatility and adaptability of identifying different types of seals, thereby improving the accuracy of seal recognition. In addition, compared with the method in the prior art of first flattening the curved text into horizontal text and then performing text recognition, this specification directly inputs the curved text into the curved text recognition model for recognition, avoiding the problems of text breakage and cumulative errors that occur during the straightening process, thereby improving the accuracy of text recognition.
[0058] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail in conjunction with the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0060] Figure 1 A schematic diagram showing an implementation environment of a seal recognition method in some embodiments of this specification;
[0061] Figure 2 A schematic diagram showing a flow chart of a seal recognition method in some embodiments of this specification;
[0062] Figure 3 A schematic diagram of a process for determining the seal coordinate information in the seal image to be identified in some embodiments of this specification is shown;
[0063] Figure 4 A schematic diagram of a process of determining an initial seal text area and a corresponding text type according to the seal slice image to be identified and the seal category in some embodiments of this specification is shown;
[0064] Figure 5 A flowchart of using the PAN++ model to detect text areas on a stamp slice image in some embodiments of this specification is shown;
[0065] Figure 6 A schematic diagram of a process of using the PCA algorithm to perform angle correction on the initial seal text area in some embodiments of this specification is shown;
[0066] Figure 7 The following is a diagram showing the architecture of the TrOCR recognition model in some embodiments of this specification;
[0067] Figure 8 Another schematic diagram of a flow chart of a seal recognition method in some embodiments of this specification is shown;
[0068] Fig. 9 Another schematic diagram of a flow chart of a seal recognition method in some embodiments of this specification is shown;
[0069] Fig.10 A schematic diagram of the structure of a seal recognition device in some embodiments of this specification is shown;
[0070] Fig.11 A schematic diagram of the structure of a computer device in this specification is shown.
[0071] Description of the accompanying symbols:
[0072] 101. Terminal;
[0073] 102. Server side;
[0074] 1001. Get module;
[0075] 1002. Seal positioning and classification module;
[0076] 1003, seal slicing module;
[0077] 1004. Text region determination and classification module;
[0078] 1005. Text angle correction module;
[0079] 1006. Text recognition module;
[0080] 1102. Computer equipment;
[0081] 1104. Processor;
[0082] 1106. Memory;
[0083] 1108, driving mechanism;
[0084] 1110, input / output module;
[0085] 1112. Input device;
[0086] 1114. Output device;
[0087] 1116. Presentation equipment;
[0088] 1118. Graphical user interface;
[0089] 1120, network interface;
[0090] 1122. Communication link;
[0091] 1124. Communication bus. DETAILED DESCRIPTION
[0092] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0093] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0094] like Figure 1 Shown is a schematic diagram of the implementation environment of a seal recognition method according to an embodiment of the present specification, which may include: a terminal 101 and a server 102. The terminal 101 and the server 102 communicate with each other through a network. The network may include a local area network (LAN), a wide area network (WAN), the Internet or a combination thereof, and is connected to a website, a user device (such as a computing device) and a back-end system.
[0095] The terminal 101 is used to collect the seal image to be identified and send the seal image to be identified to the server 102. In an optional embodiment, the terminal 101 may include but is not limited to electronic devices such as self-service terminal devices, desktop computers, tablet computers, laptop computers, smart wearable devices, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc. Of course, the terminal 101 is not limited to the above-mentioned electronic devices with a certain entity, and it can also be software running in the above-mentioned electronic devices.
[0096] The server end 102 can be used to obtain a seal image to be identified; and to determine the seal coordinate information and seal category in the seal image to be identified; and to cut the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified; and to determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category; and to perform angle correction on the initial seal text area according to the text type to obtain a first seal text area; and to obtain a pre-trained text recognition model corresponding to the text type, and use the text recognition model to identify the first seal text area to obtain a first seal text recognition result. Optionally, the server end 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms.
[0097] It should be noted that Figure 1 This is only one implementation environment of the seal recognition method provided in the embodiment of the present application. In practical applications, other implementation environments may also be included.
[0098] Figure 2 This is a flowchart of a seal recognition method provided in an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in the order of the method shown in the embodiment or the drawings or in parallel. Specifically, Figure 2 As shown, the method includes:
[0099] S201: Acquire a seal image to be identified;
[0100] S202: Determine the seal coordinate information and seal category in the seal image to be identified;
[0101] S203: performing a cutting process on the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified;
[0102] S204: determining an initial seal text area and a corresponding text type according to the seal slice image to be identified and the seal category;
[0103] S205: performing angle correction on the initial seal text area according to the text type to obtain a first seal text area;
[0104] S206: Acquire a pre-trained text recognition model corresponding to the text type, and use the text recognition model to recognize the first seal text area to obtain a first seal text recognition result.
[0105] The embodiment of this specification obtains a slice image containing only the seal through the seal coordinate information of the seal image to be identified, and determines the seal text area and the corresponding text type according to the seal category, and adopts different correction methods for different text types to perform angle correction, thereby reducing the influence of the text direction on the seal text recognition. Furthermore, considering the structural differences between different types of texts, the embodiment of this specification selects different text recognition models for different text types for text recognition, which improves the versatility and adaptability of identifying different types of seals, thereby improving the accuracy of seal recognition. In addition, compared with the method in the prior art of first flattening the curved text into horizontal text and then performing text recognition, this specification directly inputs the curved text into the curved text recognition model for recognition, avoiding the problems of text breakage and cumulative errors that occur during the straightening process, thereby improving the accuracy of text recognition.
[0106] After obtaining the seal image to be identified, the entire image needs to be detected to locate the specific position of the complete seal in the image. In order to improve the detection and recognition accuracy of subsequent steps, this step needs to detect the complete seal, which will involve the detection of seals of different colors and shapes. Therefore, it is necessary to select an algorithm with universal detection characteristics. In some embodiments of this specification, the CenterNet model is used, which can accurately detect the center point of the seal and then completely locate the seal, which can ensure high accuracy. Figure 3 In step S202, the step of determining the seal coordinate information in the seal image to be identified includes:
[0107] S301: Inputting the seal image to be identified into a pre-trained seal detection model to obtain the coordinates of the center point of the rectangular area where the seal is located;
[0108] S302: Determine seal coordinate information according to the center point coordinates and the length and width of the rectangular area.
[0109] It can be understood that when the CenterNet model detects a seal in the seal image to be identified, it will mark the area where the seal is located by generating a rectangular frame, and generate the center coordinates of the rectangular frame at the same time. The seal coordinate information can be determined based on the center coordinates and length and width values of the rectangular frame. The seal coordinate information includes the coordinates of the upper left corner and lower right corner of the rectangular area where the seal is located.
[0110] In some embodiments, determining the seal category in the seal image to be identified includes:
[0111] The seal category in the seal image to be identified is identified based on a pre-trained seal classification model, wherein the seal classification model is trained using historically collected seal images and seal category labels thereof.
[0112] Specifically, the VAN model is used in this specification for seal classification. VAN is a visual attention network with attention mechanism and large kernel convolution based on Transformer structure. It combines the advantages of attention mechanism and large kernel convolution, and captures long-range relationships by decomposing a large kernel convolution operation; and its network architecture is simpler and more efficient, with fewer parameters and computational complexity compared to other classification models. The seal categories in this specification include round seals, oval seals, square seals, diamond seals and triangular seals, etc. Among them, because the appearance of round seals and oval seals is similar, and in perspective and other situations, round seals will become oval seals, so they can be classified into one category, and polygonal seals with beveled edges such as diamond seals and triangular seals can also be classified into one category.
[0113] Furthermore, in the embodiment of this specification, in step S203, the seal image to be identified is cut according to the seal coordinate information to obtain a seal slice image to be identified, and any seal slicing tool can be used to cut the seal according to the coordinates of the upper left corner and the lower right corner of the rectangular area where the seal is located.
[0114] Further, in the embodiments of this specification, reference is made to Figure 4 As shown, in step S204, the initial seal text area and the corresponding text type are determined according to the seal slice image to be identified and the seal category, including:
[0115] S401: Inputting the seal slice image to be identified into a pre-trained text detection model to obtain an initial seal text area.
[0116] Specifically, the classified seal slice image is input into the text detection model to detect the text area. In order to ensure the accuracy of subsequent seal text recognition, and considering the problems that the seal may contain special format text, shapes (such as Chinese and English, double circles) or unusually complex backgrounds, the embodiment of this specification uses the PAN++ model to detect the text area of the seal slice image. The specific process is as follows: Figure 5As shown. The PAN++ model uses a representation method for text of any shape based on the text kernel (i.e., the central area), which can better distinguish adjacent texts, and is optimized for irregularly shaped texts, and can accurately identify and locate texts of various shapes. The PAN++ model uses a lightweight ResNet18 as the backbone network to improve the inference speed, and uses a stacked feature pyramid enhancement module (FPEM) to enhance the extracted features. Finally, a lightweight detection head that collaborates with pixel aggregation (PA) can accurately detect the text area and obtain the pixel-level position information of the seal text area. In the process of training the PAN++ model, in order to further improve the model detection performance, the diversity of training data can be enhanced through data enhancement. The optional data enhancement methods include: random horizontal flipping, random angle rotation, random cropping and filling, and random changes in brightness, contrast, saturation, etc.
[0117] S402: Calculate the curvature of the initial seal text area.
[0118] For the detected initial seal text area, the text area contour can be obtained through its pixel-level position information, a series of points on the contour are selected, and the curvature of each point is calculated, and then the curvature average value is calculated. The curvature average value is used as a measure to characterize the curvature of the initial seal text area. The larger the curvature average value, the greater the curvature of the text area.
[0119] S403: If the curvature is greater than a preset threshold, and the seal type corresponding to the initial seal text area is circular or elliptical, determining that the text type is curved text;
[0120] S404: If the curvature is less than a preset threshold, and the seal type corresponding to the initial seal text area is non-circular or non-elliptical, determine that the text type is non-curved text, wherein the non-curved text includes horizontal text, vertical text and inclined text.
[0121] Furthermore, in the embodiment of this specification, in step S205, the angle correction of the initial seal text area according to the text type includes:
[0122] If the text type is curved text, the PCA (principal component analysis) algorithm is used to perform angle correction on the initial seal text area;
[0123] If the text type is non-curved text, the initial seal text area is angle-corrected using a minimum circumscribed polygon.
[0124] In practical applications, since the angle of the seal can be arbitrary, in order to ensure the recognition efficiency and accuracy of the subsequent text recognition model, this manual requires that all text areas input to the text recognition model be corrected to the positive direction, where curved text is angle-corrected using algorithmic technology, and non-curved text is angle-corrected using the minimum circumscribed polygon method. By using the PCA algorithm to correct the angle of curved text, the main direction of the curved text can be determined based on the calculated principal components, and noise can be suppressed during the extraction of the principal components, reducing the impact of noise on the correction results, thereby ensuring the accuracy of the correction. Figure 6 As shown, the PCA algorithm is used to perform angle correction on the initial seal text area, including:
[0125] S501: Acquire coordinate information of the initial seal text area, and extract a seal text image according to the coordinate information.
[0126] Specifically, the curved text is stripped from the seal slice according to the coordinate information of the initial seal text area, and is merged with the pure white background to generate a seal text image.
[0127] S502: performing debinarization processing on the seal text image to obtain a debinarized seal text image.
[0128] Specifically, after the seal text image is debinarized, the curved text area is adjusted to pure white and the background is adjusted to pure black.
[0129] S503: Acquire the coordinates of all non-zero pixels in the debinarized seal text image.
[0130] Since debinarization only changes the color of the pixel without changing the brightness value or the number of the pixel, the coordinates of the non-zero pixels in the debinarized seal text image are the same as the coordinates of the black pixels (ie, the text area) in the original seal text image.
[0131] S504: Calculate the eigenvalues and eigenvectors of the non-zero pixel coordinates, and select the eigenvector with the largest eigenvalue as the principal component.
[0132] S505: Calculate the angle between the principal component and the horizontal coordinate axis, and use the angle as the text rotation angle of the seal text image.
[0133] By calculating the angle between the principal component and the horizontal coordinate axis, the main direction of the curved text can be determined.
[0134] S506: performing angle correction on the initial seal text area according to the text rotation angle.
[0135] In practical applications, the text rotation angle of the curved text may have a 180-degree error. At this time, it can be determined whether a secondary correction is required by judging the number of white pixels in the upper and lower parts of the seal text image in the above step S502. If the number of white pixels in the upper image is greater than the number of white pixels in the lower image, it indicates that the text rotation of the curved text is correct and no processing is required; if the number of white pixels in the upper image is less than the number of white pixels in the lower image, it indicates that the text rotation of the curved text is correct, there is a 180-degree error, and a secondary correction is required by the method of (180-text selection angle).
[0136] In some embodiments, the step of using a minimum bounding rectangle to perform angle correction on the initial seal text area includes:
[0137] Extracting the outline of the non-curved text according to the coordinate information of the initial seal text area;
[0138] Use functions in OpenCV and other libraries (such as cv2.minAreaRect) to calculate the minimum enclosing rectangle of the contour. The cv2.minAreaRect function returns a RotatedRect object, which contains the center point, width, height, and rotation angle of the minimum enclosing rectangle.
[0139] Angle correction is performed on the non-curved text area according to the rotation angle.
[0140] Furthermore, in the embodiment of this specification, in step S206, obtaining a pre-trained text recognition model corresponding to the text type, and using the text recognition model to recognize the first seal text area to obtain the first seal text recognition result further includes:
[0141] When the first seal text area is curved text, the first seal text area is recognized by using a pre-trained curved text recognition model; when the first seal text area is non-curved text, the first seal text area is recognized by using a pre-trained regular text recognition model to obtain a first seal text recognition result. Specifically, the regular text recognition model is a CRNN text recognition model specially trained for horizontal and vertical texts, and the curved text recognition model is a TrOCR model.
[0142] Compared with other curved text recognition models, the TrOCR model can directly recognize the curved text without straightening the curved text, so no cumulative error will be generated, and the model is based on semantic text recognition with high recognition accuracy. In the embodiment of this specification, the TrOCR model includes a preprocessing module, a mapping module, an image transformer and a text transformer, and the text recognition model is used to recognize the first seal text area to obtain the first seal text recognition result, including:
[0143] Inputting the first seal text area into the preprocessing module to obtain a minimum bounding rectangle and a number of text blocks uniformly divided by the minimum bounding rectangle;
[0144] Inputting the text block into the mapping module for linear mapping and position embedding to obtain a text block embedding vector;
[0145] Inputting the text block embedding vector into the image Transformer to extract features and obtain feature representation;
[0146] The feature representation is input into the text transformer for feature decoding to obtain a first seal text recognition result.
[0147] In order to solve the problem that TrOCR is prone to overfitting when the amount of training data is small, the training data needs to be expanded during the training process of TrOCR. Specifically, by extracting characters from the general Chinese character library, a large number of seal slices are generated by randomly synthesizing seal slices. The generated seal slices need to contain all Chinese characters, otherwise if a certain character is missing, the TrOCR recognition model may have recognition errors. In addition, for some seal slices that are difficult to recognize because of too light color or blurred text area, data enhancement is required. The specific implementation method is to synthesize difficult example data by analyzing specific difficult examples such as the above, and synthesize difficult example data through some image enhancement methods, so that the TrOCR model has more sufficient training on these difficult examples, thereby improving the recognition accuracy of these difficult examples. The specific image enhancement method includes: color enhancement, fuzzy text generation and covered text generation. Therefore, through the above-mentioned processing of training data, the problem of TrOCR being prone to overfitting when the amount of training data is small can be avoided in the reasoning process, and the recognition accuracy of seal texts that are difficult to recognize such as too light color, blurred text area or covered text area can be improved.
[0148] Figure 7 The architecture diagram of the TrOCR recognition model is shown in Figure 1. First, the minimum bounding rectangle of the first seal text area is generated, and then the bounding rectangle is divided into several text blocks with m rows and n columns, such as Figure 7In this example, m=2, n=4, and then all text blocks are flattened to obtain a one-dimensional vector, where the position order of the one-dimensional vector is sorted from left to right and from top to bottom. The flattened text block is linearly mapped to form a first embedding vector, and the position embedding information is added to the first embedding vector to retain the position information of the text block in the original image. The text block embedding vector formed by the first embedding vector and the position embedding information is input into the pre-trained image transformer, and after multiple self-attention layers and feedforward neural network layers, the feature representation of the image is generated, and the feature representation contains the semantic information and position information of the text. Finally, the feature representation is input into the pre-trained text transformer. Through the self-attention mechanism and the encoder-decoder attention mechanism, the text transformer can pay attention to the key information in the feature representation and the corresponding position information, thereby generating accurate text output.
[0149] In order to further improve the accuracy of seal text recognition results, in some embodiments of this specification, reference Figure 8 , the seal recognition method further includes:
[0150] S207: matching the first seal text recognition result with the fixed fields in the pre-built fixed field library to obtain the corresponding matching confidence;
[0151] S208: If the matching confidence is greater than a preset threshold, the first seal text recognition result is corrected according to the fixed field, and the corrected first seal text recognition result is determined as the final seal text recognition result.
[0152] It can be understood that in actual application scenarios, some contents in the seal are fixed. For such fixed field contents, it is first necessary to collect the relevant field contents in various application scenarios to form a fixed field library. After using the seal text recognition model to perform seal text recognition, the recognition results are automatically revised and supplemented according to the matching degree with the corresponding fixed fields in the fixed field library to ensure that the recognition of this part of the field is correct. The specific implementation method is to collect fixed fields in some application scenarios and build a fixed field library. For example, Shanghai *** Bank, Beijing *** Co., Ltd., etc. are all fixed fields registered in the industrial and commercial administration. Such fields can be collected through the Internet or public data sets. After recognition using the corresponding text recognition model, the matching confidence of the recognized text and the fields in the fixed field library is calculated. If the matching confidence of a fixed field in the fixed field library is greater than the preset threshold, the field can be used to revise and supplement the corresponding part of the recognition result, thereby improving the accuracy of the recognition of this part of the field.
[0153] In other embodiments of this specification, in order to further ensure the reliability and accuracy of the seal text recognition results, refer to Fig. 9 , the seal recognition method further includes:
[0154] S209: vertically flipping the first seal text area to obtain a second seal text area;
[0155] S210: Inputting the second seal text area into the corresponding pre-trained text recognition model for text recognition to obtain a second seal text recognition result;
[0156] S211: Calculating the confidences of the first seal text recognition result and the second seal text recognition result, and determining the seal text recognition result with a higher confidence as the final seal text recognition result.
[0157] It can be understood that when the PCA algorithm and the minimum circumscribed rectangle are used to correct the angle of the text area, the text direction may be mistakenly turned to the reverse direction, thereby affecting the recognition effect. In order to solve this problem, when the seal text recognition model is used for recognition, the positive and negative images of the seal text area (i.e., flipped 180 degrees up and down) are input into the seal text recognition model for recognition, and the final seal text recognition result is determined based on the confidence of the two recognition results. In this way, the problem of inaccurate recognition caused by angle correction can be avoided.
[0158] In some embodiments, the final text recognition result can also be determined by fusing the recognition results of models of different epochs to improve the accuracy of text recognition. Specifically, when training the text recognition model, the models trained using different epochs are saved, and then when performing text recognition, the text models of these different epochs are used to perform text recognition respectively to obtain multiple text recognition results. At this time, the final text recognition result can be determined based on the voting principle or the highest confidence principle.
[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the acquisition, storage, use, and processing of data in the technical solutions described in the embodiments of this application are in compliance with relevant regulations.
[0160] Based on the seal recognition method described above, the embodiment of this specification also provides a seal recognition device. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and combined with the necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0161] Specifically, Fig.10 This is a schematic diagram of the module structure of an embodiment of a seal recognition device provided in the embodiment of this specification, referring to Fig.10 As shown, a seal recognition device provided in an embodiment of this specification includes:
[0162] An acquisition module 1001 is used to acquire a seal image to be identified;
[0163] The seal positioning and classification module 1002 is used to determine the seal coordinate information and seal category in the seal image to be identified;
[0164] A seal slicing module 1003 is used to perform a slicing process on the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified;
[0165] The text area determination and classification module 1004 is used to determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category;
[0166] A text angle correction module 1005 is used to perform angle correction on the initial seal text area according to the text type to obtain a first seal text area;
[0167] The text recognition module 1006 is used to obtain a pre-trained text recognition model corresponding to the text type, and use the text recognition model to recognize the first seal text area to obtain a first seal text recognition result.
[0168] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.
[0169] Reference Fig.11As shown, based on the above-mentioned method, a computer device 1102 is also provided in an embodiment of the present specification, wherein the above-mentioned method is run on the computer device 1102. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes an associated instruction stored in any memory or combination of memories, the computer device 1102 may perform any operation of the associated instruction. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0170] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.
[0171] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0172] Corresponding to Figures 1 to 9 In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0173] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 9 The method shown.
[0174] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 9 The method shown.
[0175] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.
[0176] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.
[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.
[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0179] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.
[0181] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0183] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.
Claims
1. A seal recognition method, characterized in that: The method comprises: Obtaining a seal image to be identified; Determining the seal coordinate information and seal category in the seal image to be identified; Cutting the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified; Determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category; Performing angle correction on the initial seal text area according to the text type to obtain a first seal text area; A pre-trained text recognition model corresponding to the text type is obtained, and the first seal text area is recognized using the text recognition model to obtain a first seal text recognition result.
2. The method according to claim 1, characterized in that The step of determining the seal coordinate information in the seal image to be identified includes: Inputting the seal image to be identified into a pre-trained seal detection model to obtain the coordinates of the center point of the rectangular area where the seal is located; The seal coordinate information is determined according to the center point coordinates and the length and width values of the rectangular area.
3. The method according to claim 1, characterized in that The step of determining the seal category in the seal image to be identified comprises: The seal category in the seal image to be identified is identified based on a pre-trained seal classification model, wherein the seal classification model is trained using historically collected seal images and seal category labels thereof.
4. The method according to claim 1, characterized in that The step of determining the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category includes: Inputting the seal slice image to be identified into a pre-trained text detection model to obtain an initial seal text area; Calculating the curvature of the initial seal text area; If the curvature is greater than a preset threshold, and the seal type corresponding to the initial seal text area is circular or elliptical, determining that the text type is curved text; If the curvature is less than a preset threshold and the seal type corresponding to the initial seal text area is non-circular or non-elliptical, the text type is determined to be non-curved text, wherein the non-curved text includes horizontal text, vertical text and inclined text.
5. The method according to claim 4, characterized in that The step of performing angle correction on the initial seal text area according to the text type comprises: If the text type is curved text, the PCA algorithm is used to perform angle correction on the initial seal text area; If the text type is non-curved text, the initial seal text area is angle-corrected using a minimum circumscribed polygon.
6. The method according to claim 5, characterized in that The method of using the PCA algorithm to perform angle correction on the initial seal text area includes: Acquire the coordinate information of the initial seal text area, and extract the seal text image according to the coordinate information; Performing a debinarization process on the seal text image to obtain a debinarized seal text image; Obtaining all non-zero pixel coordinates in the debinarized seal text image; Calculating the eigenvalues and eigenvectors of the non-zero pixel coordinates, and selecting the eigenvector with the largest eigenvalue as the principal component; Calculating the angle between the principal component and the horizontal coordinate axis, and using the angle as the text rotation angle of the seal text image; Angle correction is performed on the initial seal text area according to the text rotation angle.
7. The method according to claim 4, characterized in that The text recognition model of the curved text includes a preprocessing module, a mapping module, an image transformer and a text transformer; The step of using the text recognition model to recognize the first seal text area to obtain a first seal text recognition result includes: Inputting the first seal text area into the preprocessing module to obtain a minimum bounding rectangle and a number of text blocks uniformly divided by the minimum bounding rectangle; Inputting the text block into the mapping module for linear mapping and position embedding to obtain a text block embedding vector; Inputting the text block embedding vector into the image Transformer to extract features and obtain feature representation; The feature representation is input into the text transformer for feature decoding to obtain a first seal text recognition result.
8. The method according to claim 1, characterized in that The method further comprises: Matching the first seal text recognition result with the fixed fields in the pre-built fixed field library to obtain the corresponding matching confidence; If the matching confidence is greater than a preset threshold, the first seal text recognition result is corrected according to the fixed field, and the corrected first seal text recognition result is determined as the final seal text recognition result.
9. The method according to claim 1, characterized in that: The method further comprises: Vertically flipping the first seal text area to obtain a second seal text area; Inputting the second seal text area into the corresponding pre-trained text recognition model for text recognition to obtain a second seal text recognition result; The confidence levels of the first seal text recognition result and the second seal text recognition result are calculated, and the seal text recognition result with a higher confidence level is determined as the final seal text recognition result.
10. A seal recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring the seal image to be identified; A seal positioning and classification module, used to determine the seal coordinate information and seal category in the seal image to be identified; A seal slicing module, used for cutting the seal image to be identified according to the seal coordinate information to obtain a seal slice image to be identified; A text area determination and classification module, used to determine the initial seal text area and the corresponding text type according to the seal slice image to be identified and the seal category; A text angle correction module, used for performing angle correction on the initial seal text area according to the text type to obtain a first seal text area; The text recognition module is used to obtain a pre-trained text recognition model corresponding to the text type, and use the text recognition model to recognize the first seal text area to obtain a first seal text recognition result.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
13. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Document image key information extraction and matching method based on graph neural network
CN112464781A
Seal recognition method and system based on deep learning and storage medium
CN113901883A
Character recognition method and device for seal
CN115482536A
Seal curved text recognition method and system, electronic equipment and storage medium
CN117274999A
Seal identification method and device, electronic equipment and storage medium
CN118196812A
Cited By
Seal identification method, system and equipment based on generative seal identification model and storage medium
CN120163970A
Image-text report generation method and device and storage medium
CN120448536A
