Investment amount category table recognition method based on convolutional neural network

Abstract

The application discloses a kind of investment amount table identification method based on convolutional neural network, method includes the following steps: table identification stage and reverse reasoning stage;Table identification stage process as follows: pdf file is converted into picture, and the picture is input to pre-trained table identification model, obtain the table area in pdf file;The component of table is identified;According to the cell in table and the content in cell are identified;The table is restored using identification and the content in cell;Reverse reasoning stage process as follows: by the picture visual features corresponding to table, the x, y coordinates corresponding to table line are calculated, compare this result with previous result, when difference exceeds preset threshold value, consider that table identification stage result is inaccurate, adjust training parameter and retrain table identification model.The application has beneficial effects: improve the precision of table identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of table recognition, in particular to a method for recognizing investment amount type table based on convolutional neural network. BACKGROUND

[0002] Table is a structured form of information expression, and PDF is portable document format. In real life, the two are often combined to express data with high similarity and high structure.

[0003] The previous technology mainly relies on image processing technology to recognize tables. The PDF file is converted into a picture, and then the features of the image to be recognized are determined manually, or the picture is processed through morphological opening or closing operation, or the image is scanned using a specific kernel function, so as to accurately locate the coordinates of each cell and table, and then recognize the cell content.

[0004] Other ideas include directly ignoring the table lines, recognizing all character content in the picture, and then performing aggregation operation on the characters to determine the position of the cells in the table according to the obtained phrases.

[0005] The above schemes are all based on image processing technology to recognize tables. Compared with the scheme based on neural network and supplemented by image processing technology, the accuracy is relatively insufficient. In addition, the previous technology does not pay enough attention to the case of cross-page table, resulting in low recognition rate of the table with cross-page. Finally, when the table in the PDF is obtained by scanning, and the scanned table is skewed, the scheme for calculating the coordinates of the cells is not ideal in recognizing the table. SUMMARY

[0006] In order to solve the technical problems of inaccurate table recognition and inability to cope with individual special cases of the existing method, the present application proposes a method for recognizing investment amount type table based on convolutional neural network, which specifically includes the following steps:

[0007] Table recognition stage and reverse reasoning stage;

[0008] The table recognition stage is used to restore the table in the PDF file;

[0009] The reverse reasoning stage is used to judge whether the restoration result of the table recognition stage is accurate;

[0010] The table recognition stage process is as follows:

[0011] S101, convert the PDF file into a picture, and input the picture into a pre-trained table recognition model to obtain the table region in the PDF file;

[0012] S102, identifying components of the table in the table area;

[0013] S103, obtaining cells in the table according to the identification, and identifying content in the cells;

[0014] S104, restoring the table by using the identification and the content in the cells;

[0015] The reverse reasoning stage process is as follows:

[0016] By the picture visual features corresponding to the table, the x and y coordinates corresponding to the table lines are calculated, and the result is compared with the result in step S1; when the difference exceeds a preset threshold, it is considered that the table recognition stage result is inaccurate, the training parameters are adjusted to retrain the table recognition model, otherwise, it is considered that the table recognition stage result is accurate.

[0017] A storage device stores instructions and data for implementing a convolutional neural network-based investment amount type table recognition method.

[0018] A convolutional neural network-based investment amount type table recognition device includes a processor and the storage device; the processor loads and executes the instructions and data in the storage device to implement a convolutional neural network-based investment amount type table recognition method.

[0019] The present application provides the beneficial effect of: improving the accuracy of table recognition. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a method flowchart of the present application;

[0021] Figure 2 is a hardware device working schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described below with reference to the drawings.

[0023] Please refer to Figure 1 , Figure 1 is a method flowchart of the present application.

[0024] The present application provides a convolutional neural network-based investment amount type table recognition method, including the following steps:

[0025] The table recognition stage and the reverse reasoning stage;

[0026] The table recognition stage is used for restoring the table in the pdf file;

[0027] The reverse reasoning stage is configured to determine whether the reduction result of the table recognition stage is accurate.

[0028] The table recognition stage is configured as follows:

[0029] S101, converting a pdf file into an image, and inputting the image into a pre-trained table recognition model to obtain a table region in the pdf file;

[0030] The training process of the table recognition model is configured as follows:

[0031] S201, obtaining a pdf file of an investment amount table, and converting the pdf file into an image;

[0032] As an embodiment, for a pdf file containing an investment amount table, color features do not provide much help for identifying where the table is located. Secondly, for the sample size, the present application refers to the size of an image obtained by converting an A4 paper word document into a pdf file and then converting the pdf file into the image. Therefore, the input sample is a 2480x3580 resolution grayscale image.

[0033] S202, labeling components of the table in the image to form a sample set;

[0034] As an embodiment, in the present application, the table name, table header, table annotation and cell on the image are all framed with a rectangular frame, and then each rectangular frame is added with category information. The final effect is that one image corresponds to one txt file, and each line in the txt file contains the coordinates of the rectangular frame and the category thereof.

[0035] S203, dividing the sample set into a training set and a test set;

[0036] In the present application, the samples are randomly divided into the training set and the test set according to a ratio of 8:2.

[0037] S204, constructing a table recognition model based on a VGG neural network;

[0038] As an embodiment, the table recognition model in the present application adopts a neural network similar to the VGG (Visual Geometry Group) mode. Specifically, the neural network is constructed in the following manner: convolutional layer (with 64 filters) - convolutional layer (with 64 filters) - pooling layer - convolutional layer (with 128 filters) - convolutional layer (with 128 filters) - pooling layer - convolutional layer (with 256 filters) - convolutional layer (with 256 filters) - convolutional layer (with 256 filters) - pooling layer - pooling layer - convolutional layer (with 512 filters) - convolutional layer (with 512 filters) - convolutional layer (with 512 filters) - pooling layer - pooling layer - convolutional layer (with 512 filters) - convolutional layer (with 512 filters) - convolutional layer (with 512 filters) - pooling layer (followed by vector expansion) - fully connected layer - softmax layer, wherein all convolutional layers use filters with a size of 3x3, stride = 1x1, and padding = 0; and all pooling layers have a kernel size of 2x2, stride = 2, and padding = 2.

[0039] S205, training the table recognition model using the training set and the test set to obtain a trained table recognition model.

[0040] The samples and the test set are respectively input into the neural network model constructed as described above to obtain the recognition results of the model on the training set and the test set.

[0041] S102, identifying the components of the table within the table region;

[0042] It should be noted that the identification here is mainly to determine which content within the inferred region is a component of the table (such as the table header, the cell, etc.), and to identify these components.

[0043] For the table to be recognized, the following assumptions are made in advance:

[0044] The table is in order and continuous. That is, the first half of the table does not appear on the second page, and the second half does not appear on the first page; nor does the first half of the table appear on the first page and the second half appear on the third page.

[0045] The picture is directly input into the initially trained table recognition model in the form of three channels, and the result is converted into corresponding coordinate points to obtain the corresponding coordinates of each cell in the table in the picture.

[0046] When all the tables in the entire pdf are acquired, the cross-page determination of the table is performed. When it is detected that there is a table in both of the two continuous pages, the determination is performed according to the following description.

[0047] First, the following rules are made by using the characteristics of the table to quickly filter the non-cross-page table:

[0048] (1) The number of columns of the tables in the two pages is inconsistent.

[0049] (2) The distance between the y coordinate corresponding to the bottom position of the table in the first page and the y coordinate corresponding to the top of the table in the second page and the margin of the page is more than 20% of the height of the entire page.

[0050] Then, according to the following rules, it is determined whether the table is cross-page:

[0051] (1) The first page table has a table header, and the second page table does not have a table header.

[0052] (2) According to the user configuration, the first page table and the second page table both have a table header.

[0053] When it is determined that the table exists in the cross-page, the front and rear tables are marked for subsequent merging.

[0054] S103, acquire the cells in the table according to the mark, and identify the content in the cells;

[0055] Step S103 is specifically as follows:

[0056] S401, establish a text area detection model and an optical character recognition model;

[0057] As an embodiment, for text region detection, a convolutional neural network is constructed, an RPN (Region Proposal Network) is introduced in the network, and finally the result is output through softmax. Specifically, the neural network is constructed in the following manner: convolutional layer (filter number is 64) - convolutional layer (filter number is 64) - pooling layer - convolutional layer (filter number is 128) - convolutional layer (filter number is 128) - pooling layer - convolutional layer (filter number is 256) - convolutional layer (filter number is 256) - convolutional layer (filter number is 256) - pooling layer - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - pooling layer - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - RPN - softmax layer, wherein the filter size used by all convolutional layers is 3x3, stride = 1x1, padding = 0; the kernel size of all pooling layers is 2x2, stride = 2, padding = 2.

[0058] For OCR, a neural network of VGG mode is constructed, and specifically, the neural network is constructed in the following manner: convolutional layer (filter number is 64) - convolutional layer (filter number is 64) - pooling layer - convolutional layer (filter number is 128) - convolutional layer (filter number is 128) - pooling layer - convolutional layer (filter number is 256) - convolutional layer (filter number is 256) - convolutional layer (filter number is 256) - pooling layer - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - pooling layer - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - convolutional layer (filter number is 512) - pooling layer (followed by vector expansion) - fully connected layer - softmax layer, wherein the filter size used by all convolutional layers is 3x3, stride = 1x1, padding = same padding; the kernel size of all pooling layers is 2x2, stride = 2, padding = 2.

[0059] S402, respectively train the text region detection model and the optical character recognition model using the data set, and obtain the trained text region detection model and the optical character recognition model;

[0060] It should be noted that the corresponding technologies of text region detection and optical character recognition are relatively mature, which benefits from the open source data set. The data set used for training the two models is directly used for the open source text region detection and OCR character set, which is the training data corresponding to the task "ICDAR2017 Competition on Multi-lingual scene text detection and script identification" and the Chinese recognition data set "YCG09".

[0061] For text region detection, the samples of the data set are corresponding pictures in various scenes such as streets and walls, and the pictures have various language texts. The labels of the data set are the four point coordinates of the rectangular frame of the text region.

[0062] For the OCR character set, the samples of the data set are pictures randomly generated from the Chinese corpus by changing the font, size, perspective and stretching of the text, and the sample size is 280*32. The label of the data set is the text in the picture.

[0063] Of course, in the processing process, these data are parsed, and the data therein are simply guaranteed to be correct. For the text region detection task: delete the rectangular coordinates without size, reconstruct the coordinates of the rectangular coordinates with overlap, and perform basic value size checking and basic relationship checking between the coordinates; for the OCR task: check that the channel number of all pictures is the same, and ensure that each picture has corresponding text.

[0064] It should be noted that in the actual investment amount table, about 10% of the total page size of the pages is different from the other pages in the pdf, so it is necessary to process these data in advance before inputting them into the program. The pdf is converted into an image according to the size of the page itself, and then the image is uniformly scaled to 2480*3580, and the scaling multiples of the length and width are recorded for subsequent image restoration.

[0065] S403, input the cell picture to the text region detection model and the optical character recognition model, and complete the recognition of the cell text content and the optical character.

[0066] Since the character set itself has been divided into training set and test set in the ratio of 99:1, the model is also divided into sample and test set in the ratio during training. Then the sample and test set are input into the text area detection model and the optical character recognition model constructed as described above to start training, so as to obtain the two models after training, and the two trained models are used for recognition.

[0067] According to the obtained cell coordinates, the original page is cropped to obtain a picture corresponding to the cell, and then the picture is uniformly scaled to a size of 328 pixels in width.

[0068] The scaled picture is input into the text area detection model to obtain coordinates corresponding to the to-be-recognized area, and then the cell is cropped according to the coordinates, and the cropped picture is input into the OCR model to obtain the text content of the cell.

[0069] S104, restore the table by using the identifier and the content in the cell;

[0070] In step S104, the specific process of table restoration is as follows:

[0071] S501, the preset rule is as follows: there is only one table in a row, and there are no two smaller tables placed side by side in the same row;

[0072] S502, organize the table contents of each page, according to the cell coordinates, start from the top-left point, and sequentially organize the cells near this cell into the same row, and in this way, organize the cells in the whole row; then according to the top-left point, obtain the adjacent vertical cells, and then organize the cells in the whole column;

[0073] S503, according to the rule that the number of columns and the number of rows are equal after the sub-cells in the cell are merged, screen out the cells that are not recognized;

[0074] S504, fill in the cells that are not recognized to obtain the restored complete table.

[0075] It should be noted that when reconstructing the table, there is a prior assumption that there is only one table in a row. That is, there are no two smaller tables placed side by side in the same row.

[0076] Organize the table contents of each page, according to the coordinates, start from the top-left point, and sequentially organize the cells near this cell into the same row, and in this way, organize the cells in the whole row; then according to the top-left point, obtain the adjacent vertical cells, and then organize the cells in the whole column as before; similarly, organize the whole table.

[0077] After the table recognition is completed, most of the table is correctly recognized, but there are a few cases where some cells in the entire page are not recognized, that is, the model does not label some cells in the table that should be recognized, so after the recognition is completed, an additional filling process needs to be added.

[0078] For this filling process, there is a prior assumption that the table is regular.

[0079] That is, after merging the sub-cells in the cell (if there are sub-cells), the number of columns in each row is equal, and the number of rows in each column is equal.

[0080] According to this assumption, it can be deduced whether there are cells that have not been recognized, and if there are cells that have not been recognized, the program will fill in the missing cell positions in the table.

[0081] Finally, according to the scaling factor described above, the inverse of the scaling factor is taken to restore the table coordinates.

[0082] The reverse reasoning stage process is as follows:

[0083] By calculating the x, y coordinates of the table line corresponding to the picture visual features, comparing the results with the results in step S1, when the difference exceeds the preset threshold, it is considered that the table recognition stage result is inaccurate, the training parameters are adjusted to retrain the table recognition model, otherwise, it is considered that the table recognition stage result is accurate.

[0084] The specific process of the reverse reasoning stage is as follows:

[0085] S601, pre-processing the picture corresponding to the table to obtain a pre-processed picture;

[0086] S602, obtaining the x, y coordinates of the table boundary through the table region obtained in step S101, and cutting a new region after expanding the table boundary by several pixels;

[0087] S603, for the new region, summing each row and each column in the region to obtain a row array and a column array;

[0088] S604, square root of the elements of the row array and the column array, and smooth processing to obtain the peak value number of the row array and the peak value number of the column array;

[0089] S605, subtracting 1 from the peak value number of the row array and the peak value number of the column array, and comparing with the table row number and the table column number predicted by the table recognition model, if they are equal or the difference is less than a preset value, it is considered that the table recognition model prediction is accurate, otherwise it is considered that the table recognition model prediction is inaccurate.

[0090] As an embodiment, the processed picture is restored to the original picture, and the picture is scaled to 1920 pixels in width. A larger kernel function is used to perform morphological operations on the image, so as to obtain the overall position of the table.

[0091] Then the maximum bounding rectangle of the processed image is obtained. According to the rectangle, it can be calculated whether the picture has a whole deflection. If the rectangle has a deflection angle of more than 10°, the picture is subjected to an affine transformation to solve the deflection of the scanned picture.

[0092] Then the picture is slightly blurred and salt and pepper noise is removed. The picture is then converted into a gray space, and an adaptive Gaussian thresholding scheme is used to binarize the picture. Then the binarized picture is accumulated by row and by column respectively, so as to obtain one-dimensional arrays of the sum of each row and each column.

[0093] Therefore, after the coordinates of each cell are calculated by the neural network, the x and y coordinate values (four points) of the inferred boundary of the table are expanded by 50 pixels in the up, down, left and right directions. The original picture is cropped according to the region, the sum of each row and each column of the obtained region is calculated, the square root of each element of the obtained array is calculated, and then the array is smoothed. The number of peaks in the array can be calculated.

[0094] When the number of peaks obtained from the array of the sum of each row minus 1 is equal to or close to the number of rows of the table predicted by the neural network, it is considered that the number of columns of the table is successfully predicted.

[0095] Similarly, when the number of peaks obtained from the array of the sum of each column minus 1 is equal to or close to the number of columns of the table predicted by the neural network, it is considered that the number of rows of the table is successfully predicted.

[0096] See Figure 2 , Figure 2 is a hardware device working schematic diagram of an embodiment of the present application, which specifically comprises: an investment amount type table recognition device 401 based on a convolutional neural network, a processor 402 and a storage device 403.

[0097] The investment amount type table recognition device 401 based on a convolutional neural network: the investment amount type table recognition device 401 based on a convolutional neural network realizes the investment amount type table recognition method based on a convolutional neural network.

[0098] The processor 402: the processor 402 loads and executes the instructions and data in the storage device 403 to realize the investment amount type table recognition method based on a convolutional neural network.

[0099] The storage device 403 stores instructions and data, and is configured to implement the investment amount type table recognition method based on the convolutional neural network.

[0100] The application has the beneficial effect of improving the accuracy of table recognition.

[0101] The above description is only the preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for recognizing investment amount tables based on convolutional neural networks, characterized by: include: Table recognition stage and reverse reasoning stage; The table recognition stage is used to restore the table in the PDF file; The reverse reasoning stage is used to determine whether the restoration result of the table recognition stage is accurate; The process of the table recognition stage is as follows: S101, converting the PDF file into an image, and inputting the image into a pre-trained table recognition model to obtain the table area in the PDF file; S102. In the table area, mark the components of the table; S103, obtaining a cell in the table according to the identifier, and identifying the content in the cell; S104, using the identifier and the content in the cell, restore the table; The reverse reasoning process is as follows: The x and y coordinates of the table lines are calculated using the visual features of the image corresponding to the table. This result is compared with the result of the table recognition stage. If the difference exceeds a preset threshold, the result of the table recognition stage is considered inaccurate, and the training parameters are adjusted to retrain the table recognition model. Otherwise, the result of the table recognition stage is considered accurate. The specific process of the reverse reasoning stage is as follows: S601, preprocessing the image corresponding to the table to obtain a preprocessed image; S602: Obtain the x and y coordinates of the table boundary from the table area obtained in step S101, expand the table boundary by a certain number of pixels, and then crop to obtain a new area; S603: For the new area, calculate the sum of each row and column of pixels in the area to obtain a row array and a column array; S604, taking square roots of the elements of the row array and the column array, and performing smoothing processing to obtain the number of peak values ​​of the row array and the number of peak values ​​of the column array; S605. After reducing the number of peak values ​​of the row array by 1 and the number of peak values ​​of the column array by 1, the results are compared with the corresponding number of rows and columns of the table predicted by the table recognition model. If they are equal or the difference is less than a preset value, the prediction of the table recognition model is considered accurate. Otherwise, the prediction of the table recognition model is inaccurate.

2. The method for recognizing investment amount tables based on a convolutional neural network according to claim 1, characterized in that: The training process of the table recognition model is as follows: S201. Obtain a PDF file of an investment amount table and convert it into an image; S202, labeling each component of the table in the image to form a sample set; S203, dividing the sample set into a training set and a test set; S204, constructing a table recognition model based on VGG neural network; S205 : Using the training set and the test set to train the table recognition model, a trained table recognition model is obtained.

3. The method for recognizing investment amount tables based on a convolutional neural network according to claim 1, characterized in that: Step S101 is specifically as follows: S301: Input the image into a pre-trained table recognition model to obtain the coordinates of each cell in the table in the image; S302, using table features to filter non-spread tables; S303: Further cross-page determination is performed on the remaining tables, and tables that cross pages are marked for restoration when they are rebuilt.

4. The method for recognizing investment amount tables based on a convolutional neural network according to claim 3, wherein: The process of filtering non-spread-page tables in step S302 is as follows: If any of the following characteristics is met, the table is considered a non-spread table: (1) The number of columns in the tables on the two pages is inconsistent; (2) The y-coordinate corresponding to the bottom of the table on the previous page and the y-coordinate corresponding to the top of the table on the next page are more than 20% of the height of the entire page away from the page margin.

5. The method for recognizing investment amount tables based on a convolutional neural network according to claim 4, characterized in that: The process of cross-page determination in step S303 is as follows: If any of the following rules are met, it means that the table spans pages. The rules are as follows: (1) The table on the previous page has a header, but the table on the next page does not; (2) According to user configuration, both the table on the previous page and the table on the next page have headers.

6. The method for recognizing investment amount tables based on a convolutional neural network according to claim 1, wherein: Step S103 is specifically as follows: S401, establishing a text region detection model and an optical character recognition model; S402: Using the data set to train a text region detection model and an optical character recognition model, respectively, to obtain trained text region detection models and optical character recognition models; S403: Input the cell image into the text area detection model and the optical character recognition model to complete the recognition of the cell text content and the optical character.

7. The method for recognizing investment amount tables based on a convolutional neural network according to claim 1, wherein: In step S104, the specific process of table restoration is as follows: S501. The preset rule is as follows: there is only one table in a row, and no two smaller tables are placed side by side in the same row; S502: Organize the table contents of each page. Starting from the upper left corner, organize the cells adjacent to the upper left corner cell into the same row according to the cell coordinates. Continue organizing the cells of the entire row in this way. Then, based on the point in the upper left corner, get the adjacent vertical cells and organize the cells in the entire column; S503, filtering out unrecognized cells based on the rule that after merging sub-cells within a cell, the number of columns and the number of rows in each row are equal; S504: Fill in the unrecognized cells to obtain a restored complete table.

8. A storage device, characterized in that: The storage device stores instructions and data for implementing a method for recognizing investment amount tables based on a convolutional neural network as described in any one of claims 1 to 7.

9. A device for recognizing investment amount tables based on a convolutional neural network, characterized by: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a method for recognizing investment amount tables based on a convolutional neural network as described in any one of claims 1 to 7.

Images