A method for automatically reviewing images of form agreements with seals

Abstract

The invention discloses an automatic review method for a table protocol image with a seal, comprising the following steps: S1: determining whether an original image contains a seal and whether the position is correct, and if so, continuing to perform text recognition; S2: determining a text box position according to a preset text detection algorithm SegLink, and then using a preset text recognition algorithm CRNN to recognize text content; S3: fine-tuning the image capable of detecting keywords according to the angle of the text box until the horizontal frame line of the table is in a horizontal position, and then recognizing the contents of all cells of the table to determine whether they are complete; S4: judging an image that does not meet any of the above conditions as incomplete in content and failing to pass the review; S5: comparing the recognized cell content with a preset value to determine whether they are the same, and if so, recognizing through a convolutional neural network, performing polar coordinate transformation, and then comparing the preset contents to determine whether they are consistent, and if so, judging that the review is passed. The invention solves the problems of low efficiency and time-consuming of existing manual review.

Description

Technical Field

[0001] The invention relates to the technical field of text recognition and image recognition, and in particular to a method for automatically reviewing images of table protocols with seals. Background Art

[0002] In the process of authorizing corporate credit information, the agreement is a very important contractual document with legal effect. Under the existing credit system, corporate registration on the state-owned comprehensive credit platform requires filling out and submitting an agreement with the company seal according to the template. The relevant department head shall verify the company name, corporate legal person, contact number, company seal and other information. The number of companies under different prefecture-level cities in the same province is huge, so the workload of reviewing the agreement has increased accordingly. Continuing to use manual methods will result in low efficiency and long review time. Combining text recognition and seal recognition technology, the review of the agreement content can be completed directly based on the image, which greatly shortens the time expenditure and reduces labor costs. Summary of the invention

[0003] In order to solve the above problems, the present invention discloses a method for automatically reviewing a form agreement image with a seal, and the method comprises the following steps:

[0004] S1: Determine whether the original image contains a seal and whether the seal is in the correct position according to the preset seal color, shape and position. If the conditions are met, continue to perform text recognition on the agreement;

[0005] S2: Determine the position of the text box according to the preset text detection algorithm SegLink, and then use the preset text recognition algorithm CRNN to recognize the text content. Output the result according to whether the text content contains the title of the data item in the agreement. For the result that the keyword is not recognized, considering that the recognition result is not ideal due to different image acquisition methods, this method uses multiple repeated recognitions to ensure the best recognition effect. According to whether the text content contains the defined content keywords, if it does not contain the predefined content keywords, it may be because the image position is not correct, then the image is rotated 90 degrees no more than three times until the keyword can be recognized;

[0006] S3: fine-tuning the image in which the keyword can be detected according to the angle of the text box until the horizontal frame line of the table is in a horizontal position, and then identifying the contents of all cells in the table, identifying the names and contents of the required items in the table according to preset rules, and judging whether they are complete;

[0007] S4: Images that do not meet any of the above conditions are considered incomplete and will not pass the review;

[0008] S5: Check the correctness of the table with complete content, including comparing the recognized cell content with the preset value. If they are the same, the seal is recognized and intercepted through the yolov3 convolutional neural network, and polar coordinate transformation is performed. The seal content is then obtained through the preset text recognition model and compared with the preset content. If they are consistent, it is judged to have passed the review.

[0009] As an improvement of the present invention, in S1, the entire image is firstly color detected to screen out images without seals. For images that meet the requirements, the image is cut in half and then subjected to secondary color detection to determine whether the seal is in the designated area. Then, the image to be reviewed in the next step is finally determined based on the seal type corresponding to red or blue.

[0010] As an improvement of the present invention, the color detection in S1 changes the image from an RGB model to an HSV model, where H represents hue, S represents saturation, and V represents brightness. Compared with the commonly used RGB, the HSV model is more in line with the way people describe and interpret colors. According to the value range of red and blue, it is determined whether the image contains red and blue areas.

[0011] As an improvement of the present invention, the red value range is H (156-180), S (43-255), V (46-255), the blue value range is H (0-10), S (43-255), V (46-255), and the pixel threshold is 100.

[0012] As an improvement of the present invention, the color of the agreement seal in S1 is red or blue, including a round red company seal and a square blue legal person seal. In addition, according to the agreement template, important information including basic information and seals are concentrated in the upper half, and the lower half is the agreement terms, which are not within the scope of detection. Therefore, the upper half of the image is captured, and the original image is determined to contain the seal and the position is correct according to the preset seal color, shape and position.

[0013] As an improvement of the present invention, the image of the agreement is collected by taking a photo or scanning and entering it. Before the final text recognition is performed, the image is adjusted using a rotation algorithm based on the text box angle output by the detection algorithm SegLink.

[0014] As an improvement of the present invention, more than 100 pixels indicate that the image contains a red or blue area.

[0015] The beneficial effects of the present invention are as follows: the automatic review method for the image of a form agreement with a seal provided by the present invention combines text recognition and seal recognition technology to realize the completeness detection of the form content (whether it contains a seal, whether the form content is filled in completely), recognizes the text content of the form and the text content of the seal, and then completes the review of the agreement according to the rules, reviews hundreds of thousands of images in a short time, and solves the practical problems of the existing inefficient and time-consuming manual review. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flowchart of the audit method.

[0017] Figure 2 This is a schematic diagram of network structure recognition using yolov3 convolutional neural network.

[0018] Figure 3 Schematic diagram of network structure using text recognition.

[0019] Figure 4 This is an example image of an agreement template.

[0020] Figure 5 Schematic diagram of the output data of the text recognition neural network.

[0021] Figure 6 Schematic diagram of the training process of the seal recognition neural network.

[0022] Figure 7 Schematic diagram of the output data of the seal recognition neural network.

[0023] Figure 8 This is a schematic diagram of the circular seal after polar coordinate transformation. DETAILED DESCRIPTION

[0024] The following is combined with Figure 1-8 The present invention will be further illustrated with the following specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0025] Example: Figure 1 As shown, the method comprises the following steps:

[0026] S1: Determine whether the original image contains a seal and whether the seal is in the correct position according to the preset seal color, shape and position. If the conditions are met, continue to perform text recognition on the agreement;

[0027] S2: Determine the position of the text box according to the preset text detection algorithm SegLink, and then use the preset text recognition algorithm CRNN to recognize the text content, and determine whether the text content contains the defined content keyword. If it does not contain the predefined content keyword, rotate the image by 90 degrees no more than three times until the keyword can be recognized;

[0028] S3: fine-tuning the image in which the keyword can be detected according to the angle of the text box until the horizontal frame line of the table is in a horizontal position, and then identifying the contents of all cells in the table, identifying the names and contents of the required items in the table according to preset rules, and judging whether they are complete;

[0029] S4: Images that do not meet any of the above conditions are considered incomplete and will not pass the review;

[0030] S5: Check the correctness of the table with complete content, including comparing the recognized cell content with the preset value. If they are the same, the seal is recognized and intercepted through the yolov3 convolutional neural network, and polar coordinate transformation is performed. The seal content is then obtained through the preset text recognition model and compared with the preset content. If they are consistent, it is judged to have passed the review.

[0031] like Figure 4 As shown, a specific example of the original image (agreement template image) is shown as an example, which includes the following information: agreement title, authorized enterprise name, unified social credit code, legal representative name, legal representative contact number, company seal signature location, date and other corporate information. According to existing technology, it can completely replace the manual review method, and this method can be used to achieve automatic review. Among them, the shape of the company seal is specified as a red or blue circle. According to the agreement template, important information including basic information and seals are concentrated in the upper half, and the lower half is the agreement terms, which are not within the scope of detection. Therefore, the upper half of the image is intercepted, and then the original image is judged whether it contains a seal and the position is correct according to the preset seal color, shape and position.

[0032] Since the agreement images are manually taken and uploaded, there is bound to be a problem of multi-directional text boxes. Therefore, a detection algorithm SegLink that can detect text at any angle is used. The original four parameters (x, y, w, h), where the parameters x, y, w, and h refer to the horizontal coordinate of the center point, the vertical coordinate of the center point, the width of the rectangle, and the height of the rectangle, specify the position of a target, are replaced with five parameters (x, y, w, h, θ), where θ represents the rotation angle of the text box. The specific network structure is as follows: Figure 2As shown in the figure, VGG16 is used as the backbone network, and the fully connected layers (fc6, fc7) are replaced with convolutional layers (conv6, conv7), followed by four convolutional layers (conv8, conv9, conv10, conv11) to extract deeper features. The feature maps of the six layers conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11 are then convolved to obtain segments and links. In order to detect targets at multiple scales, that is, large feature maps are good at detecting small objects, while small feature maps are good at detecting large objects. The sizes of conv4 to conv11 are reduced by 1 / 2 respectively. In segment detection, each feature The number of channels of the map (feature map) after convolution is 7, of which two represent the confidence value of whether the segment is text (0,1), and the remaining five are the five offsets of the segment relative to the defaultbox at the corresponding position. In link detection, there are mainly two situations, one is intra-layer link detection and the other is cross-layer link detection. Intra-layer link detection refers to the link between each segment and its adjacent 8 segments in the same feature map, while cross-layer links are used to detect the redundancy caused by repeated detection of segments of the same text at different layers. When all segments are detected, the final text line is obtained through the fusion rules.

[0033] In order to ensure the accuracy of recognition, the detected text box is adjusted according to its angle θ, and then output to the pre-set text recognition network for recognition. The network structure is as follows: Figure 3 As shown in the figure, the network structure consists of three parts, from bottom to top: convolution layer, using CNN, which is used to extract feature sequences from the input image; recurrent layer, using RNN, which is used to predict the label (true value) distribution of the feature sequence obtained from the convolution layer; transcription layer, using CTC, which is used to convert the label distribution obtained from the recurrent layer into the final recognition result through operations such as de-reintegration. Taking the text image detected by the above method as an example, a 100*32 normalized height entry image is input, and feature maps are extracted based on 7-layer CNN. The feature maps are split into columns (Map-to-Sequence), and the 512-dimensional features of each column are input into two layers of 256-unit bidirectional LSTM for classification. During the training process, the approximate soft alignment of the character position and the class label is achieved through the guidance of the CTC loss function, and the result is finally output. Figure 5 Shown is a schematic diagram of text recognition output data.

[0034] Specifically, the present invention can identify the circumscribed square of a circular seal through a specific deep convolutional neural network model. The training images are multiple agreement images containing circular seals. The position of the circular seal in each sample is manually annotated in the form of (x, y, w, h), and the label (label) is saved in the form of an XML file. The model used in this method is yolov3, and the training flow chart is as follows: Figure 6 As shown in the figure, the process shown in the upper part is to use a convolutional neural network to extract features from the image. As the network continues to propagate forward, convolution or pooling with a stride greater than 1 is used to make the size of the feature map smaller and smaller. Each pixel will represent a more abstract feature pattern until the output feature map is reduced to 1 / 32 of the original image. The lower part describes the process of generating candidate areas. First, the original image is divided into multiple small squares, each of which is 32×32 in size. Then, a series of anchor frames are generated with each small square as the center. The entire image will be covered by the anchor frame. Based on each anchor frame, a corresponding prediction frame is generated. These prediction frames are labeled according to the positional relationship between the anchor frame and the prediction frame and the real frame of the object in the image, that is, the size of the intersection area. Finally, the feature map and the predicted label are associated, a loss function is created, and the end-to-end training process is started. Figure 7 Shown is a schematic diagram of seal recognition output data.

[0035] Specifically, yolov3 seal recognition is used to obtain the position coordinates (x, y) and radius r of the seal center point, and then the rectangular coordinate system is converted into a polar coordinate system. The text is straightened and the image is binarized after the text is straightened. The horizontal and vertical projections are performed respectively. According to the statistics of black pixels and white pixels after projection, the threshold is set to determine the position of the character. According to the position coordinates, a rectangular box is drawn, and then the pre-set text recognition model is used for text recognition.

[0036] In S1, the whole image is firstly color-detected to screen out images without stamps. For images that meet the requirements, the image is cut in half and then color-detected again to determine whether the stamp is in the specified area. Then, the image to be reviewed in the next step is finally determined based on the stamp type corresponding to red or blue. The color detection in S1 changes the image from RGB model to HSV model, where H represents hue, S represents saturation, and V represents brightness. Based on the value ranges of red and blue, it is determined whether the image contains red and blue areas. The value ranges of red are H(156-180), S(43 -255), V(46-255), the value range of blue is H(0-10), S(43-255), V(46-255), the pixel threshold is 100, and more than 100 pixels indicate that the image contains red or blue areas. The color of the agreement seal in S1 is red or blue, including a round red company seal and a square blue legal person seal. The image of the agreement is collected by taking a photo or scanning. Before the final text recognition, the image is adjusted using a rotation algorithm based on the text box angle output by the detection algorithm SegLink.

[0037] Finally, it should be noted that the embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically reviewing a form agreement image with a seal, characterized in that: The method comprises the following steps: S1: Determine whether the original image contains a seal and whether the seal is in the correct position according to the preset seal color, shape and position. If the conditions are met, continue to perform text recognition on the agreement; S2: Determine the position of the text box according to the preset text detection algorithm SegLink, and then use the preset text recognition algorithm CRNN to recognize the text content, and determine whether the text content contains the defined content keyword. If it does not contain the predefined content keyword, rotate the image by 90 degrees no more than three times until the keyword can be recognized; S3: fine-tuning the image in which the keyword can be detected according to the angle of the text box until the horizontal frame line of the table is in a horizontal position, and then identifying the contents of all cells in the table, identifying the names and contents of the required items in the table according to preset rules, and judging whether they are complete; Specifically, the detected text box is adjusted according to its angle θ, and then the pre-set text recognition network is output for recognition. The network structure includes three parts, from bottom to top: convolution layer, using CNN, which is used to extract feature sequences from the input image, recurrent layer, using RNN, which is used to predict the label distribution of the feature sequence obtained from the convolution layer, transcription layer, using CTC, which is used to convert the label distribution obtained from the recurrent layer into the final recognition result through de-reintegration operation. Taking the text image detected by the above method as an example, a 100*32 normalized height entry image is input, and a feature map is extracted based on a 7-layer CNN. The feature map is split by column, and the 512-dimensional features of each column are input into two layers of 256-unit bidirectional LSTM for classification. During the training process, the approximate soft alignment of the character position and the class label is achieved through the guidance of the CTC loss function, and finally the result is output; S4: Images that do not meet any of the above conditions are considered incomplete and will not pass the review; S5: Check the correctness of the table with complete content, including comparing the recognized cell content with the preset value. If they are the same, the seal is identified and intercepted through the yolov3 convolutional neural network, and polar coordinate transformation is performed. The seal content is obtained through the preset text recognition model, and the preset content is compared to see if it is consistent. If it is consistent, it is judged to have passed the review. In S1, the whole image is firstly color detected to filter out images without seals. For images that pass the color detection, the image is cut in half and then a secondary color detection is performed to determine whether the seal is in the specified area. Then, the image to be reviewed in the next step is finally determined according to the seal type corresponding to red or blue. The color detection in S1 changes the image from the RGB model to the HSV model, where H represents hue and S represents saturation. V represents brightness. According to the value ranges of red and blue, determine whether the image contains red and blue areas. The value range of red is H (156-180), S (43-255), V (46-255), and the value range of blue is H (0-10), S (43-255), V (46-255). The pixel threshold is 100. The color of the seal of the agreement in S1 is red or blue, including a round red company seal and a square blue legal person seal. The image of the agreement is collected by taking a photo or scanning. Before the final text recognition, the image is adjusted using a rotation algorithm based on the text box angle output by the detection algorithm SegLink. If there are more than 100 pixels, it means that the image contains red or blue areas.

Images