A method, system and medium for identifying a cell region of a table image

Abstract

The present application provides a kind of cell area identification method, system and medium of table image, the method comprises: inputting the image data of document table;With backbone network capture the image information of document table, the image information of document table is processed, and the feature information of multiple levels is obtained;With the help of feature pyramid network, the internal information features of table image are obtained by fusing the feature information of multiple levels;The internal information features of table image are reorganized based on low-rank decomposition axis attention mechanism;Based on cell space position detection branch network, the spatial position of cell is obtained;Based on cell logic information branch network, the internal information features of table image and spatial position information are fused to construct the graph network of table, and the adjacent row and column information of cell is predicted using GCN network;The mapping of the reorganized recognition information on the table image is carried out.The present application improves the recognition accuracy of table image, can provide better feature representation and better generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a cell region identification method and system for irregular table images and a medium. BACKGROUND

[0002] The market size of the entire document image intelligent recognition and analysis industry continues to grow. Industries with heavy text and document processing tasks or a large amount of historical data are embracing document image intelligent recognition and analysis, such as finance, healthcare, administration, retail, logistics, education, manufacturing, information communication, etc.

[0003] In the financial, insurance, legal and other industries, document image intelligent recognition and analysis technology can be applied to contract review, claim review, legal document processing and other fields. For example, insurance companies can use document image intelligent recognition technology to automatically review files in claim applications, improving review efficiency and accuracy; law firms can use document image intelligent analysis technology to process a large number of legal documents, improving work efficiency.

[0004] In the medical, education and other fields, document image intelligent recognition and analysis technology can be applied to medical record management, education resource management and other fields. For example, hospitals can use document image intelligent recognition technology to realize the automated management of electronic medical records, improving the level and efficiency of medical services; schools can use document image intelligent analysis technology to process education resources, such as automatic scoring, test paper analysis, etc., to improve teaching quality and efficiency.

[0005] Specifically in China, the market size of document image intelligent recognition in the financial industry has been growing year after year. The information of the underlying assets' commercial flow, physical flow, capital flow and information flow is scattered on contracts, invoices, warehouse-in and warehouse-out documents, bank statements, business licenses, etc. in paper or electronic form. Through document image intelligent recognition, each transaction voucher is automatically verified to ensure the authenticity of the transaction and the completeness of the documents.

[0006] Document image intelligent recognition and analysis involves many image types. The current industry and academic focus and difficulty is the processing of table type images in document images. Tables are different from the common text structure of document images. Tables are a combination of rows and columns with the characteristics of merged cells interference, which makes it impossible to integrate the processing of document images. Table structure recognition refers to detecting the spatial coordinates and logical relationships of the rows, columns and cells of the table, thereby identifying its structure.

[0007] Due to the low frequency of early table use, limited methods, and relatively simple table structure, simple rules can meet the detection of table regions and the extraction of structural information. Therefore, the early table detection and recognition research is mainly based on the heuristic rule method, that is, a set of rules are specified to make decisions in order to identify the table that meets the specific conditions. Since deep learning has shown outstanding performance in the field of computer image research, methods based on deep learning have gradually become the mainstream. Numerous neural network architectures in the field of computer vision are used in the detection and recognition of table images. Specifically, the following methods are used:

[0008] 1. Machine learning combined with rules: Support Vector Machine (SVM) is used in combination with rules to distinguish table regions and achieve table region detection.

[0009] 2. Convolutional network: (for example)

[0010] (1) Using the FasterR-CNN network to merge the corner and row and column information;

[0011] (2) Regression-based target detection model YOLOv3.

[0012] For the heuristic rule method, it has good effect on table image with fixed structure and small number of rows and columns, but it has the disadvantage of poor recognition effect on complex table structure, large table, and super-long cross-row table in real demand. The problems are summarized as poor generalization and complex rule design.

[0013] For the convolutional network method using deep learning, the powerful data extraction and fitting ability of the deep convolutional network improves the recognition ability of the model for complex table structure, large table, and super-long cross-row table. However, due to the insufficient design of the existing model for the representation and extraction of the connection between cells, the long training time of the model for a large amount of data, and the large number of parameters of the model, the existing model method has the problems of low accuracy and slow speed for complex merged cell images. SUMMARY

[0014] The main purpose of the present application is to provide a table image cell region recognition method, system and medium, which aims to improve the recognition accuracy of table images and provide better feature representation and better generalization ability.

[0015] In order to achieve the above purpose, the present application provides a table image cell region recognition method, which comprises the following steps:

[0016] Step S10, input the image data of the document table, set I∈i 3×H×W represents the input table image, where H and W represent the height and width of the input image, respectively;

[0017] Step S20, the image information of the document table is captured by taking ResNet50 as the backbone network, the image information of the document table is processed, and multi-level feature information is obtained.

[0018] Step S30, the multi-level feature information is fused by means of the feature pyramid network FPN to obtain the internal information feature F of the table image inter ;

[0019] Step S40, the internal information feature F of the table image is reorganized based on the low-rank decomposition axis attention mechanism inter to enhance the connection between the cells.

[0020] Step S50, the row, column and cell segmentation task of the table image is performed based on the cell spatial position detection branch network to obtain the spatial position b of the cell i ;

[0021] Step S60, the internal information feature F of the table image is fused based on the cell logical information branch network inter and the spatial position information to construct the table graph network, the GCN network is used to perceive the row and column information, and finally the adjacent row and column information l of the cell is predicted. i .

[0022] Step S70, after the spatial position information and the logical information of the table image are fused, the reorganized recognition information is mapped on the table image.

[0023] Further technical solutions of the present application are that the step S20, taking ResNet50 as the backbone network to capture the image information of the document table, processing the image information of the document table, obtaining multi-level feature information, the step includes:

[0024] The output size is 4, 8, 16, 32, f1, f2, f3, f4, respectively, representing 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image information, and these scales correspond to different levels of the pyramid FPN network, which are used for detecting targets of different sizes.

[0025] Further technical solutions of the present application are that the step S30, the multi-level feature information is fused by means of the feature pyramid network FPN to obtain the internal information feature F of the table image inter , the step includes:

[0026] The feature pyramid network FPN sequentially performs upsampling operation on the bottom feature map of the network to obtain a high-resolution feature map, and performs downsampling operation on the high-level feature map to obtain a low-resolution feature map.

[0027] Further technical solutions of the present application are that the step S40 is based on a low-rank decomposition axis attention mechanism to perform internal information feature F inter The step of performing regularization and enhancing the connection between the representation cells comprises:

[0028] Step S401, using a matrix decomposition head to perform regularization on the table two-dimensional information captured by the ResNet50 network and the feature pyramid FPN network, and enhance the connection between the representation cells;

[0029] Step S402, using a low-rank decomposition axis attention to perform table image global information regularization.

[0030] Further technical solutions of the present application are that the step of using a low-rank decomposition axis attention to perform table image global information regularization comprises:

[0031] Step S403, inputting internal features F inter For a matrix with a specification of [B, C, H, W]; during the specification regularization process, the channel number C of the input tensor is decomposed into S small matrices with a channel number of D, and then each small matrix is expanded in the spatial dimension to form a small tensor with a shape of [B*S, D, N], where N is obtained by multiplying the spatial dimensions H and W;

[0032] Step S404, initializing a base matrix bases with a size of [B*S, D, R], where R represents the rank of the base matrix; initializing a coef matrix, which is equal to the product of the regularization matrix obtained in step S403 after 1D and 2D transformation and the multiplication of the bases matrix bmm, with a size of [B*S, N, R];

[0033] Step S405, under the constraint of the number of times, constantly performing a local reasoning process to obtain the final base matrix bases and the coefficient matrix coef, wherein the maximum iteration number is constrained to be 7; the local reasoning process is as shown in formulas 1 to 4, wherein the numerator and the denominator tensors correspond to the update rules of the coef and the bases matrices, the numerator is the product of the input matrix X and the current base matrix, and the denominator is the product of the current weight matrix coef and the transposed current base matrix base;

[0034] numerator=BMM(X,base) (1)

[0035] denominator=BMM(coef,BMM(transpose(base),base)) (2)

[0036]

[0037] Step S406, multiplying the base matrix bases and the coefficient matrix coef to obtain a matrix of size [B*S, D, N];

[0038] Step S407, reorganizing the matrix of size [B*S, D, N] into a matrix of size [B, C, H, W] to obtain the result of matrix decomposition;

[0039] Step S408, transforming into an X=[B, C, W, H] matrix, and performing row-axis convolution on X to obtain a weighted tensor row of size [B, C, W, 1] att , X=X×row att ;

[0040] Step S409, performing column-axis convolution on X to obtain a weighted tensor col of size [B, C, 1, H] att , X=X×col att ;

[0041] Step S410, reorganizing X through 1×1 convolution and outputting, a matrix of size [B, C, W, H].

[0042] A further technical solution of the present application is that the step S50 includes the following steps of detecting the spatial position of the cell based on the cell spatial position detection branch network to perform the row, column and cell segmentation task on the table image to obtain the spatial position b i of the cell.

[0043] The cell spatial position detection branch network uses a 3×3 convolution kernel for decoding, and the input channel is reduced from 1024 to 256;

[0044] The row feature matrix f row ∈i 256×H , the column feature matrix f col ∈i 256×W and the cell feature matrix f cell ∈i 256×H×W are obtained through the row convolution module, the column convolution module and the cell convolution module, wherein the size of the row feature matrix is [256, image height], the size of the column feature matrix is [256, image width], and the size of the cell feature matrix is [256, image height, image width];

[0045] The convolution kernel is sampled to 64, and a 1×1 convolution kernel is used for output to obtain the predicted segmentation map y full , wherein The scale size is [category, image height, image width], and K is the predicted classification category: background, boundary, and cell;

[0046] The mapping information is output as a table segmentation mask image after post-processing by the step S70.

[0047] A further technical solution of the present invention is that, in step S60, the internal information features F of the table image are fused based on the cell logical information branch network. inter A graph network is constructed for the table based on spatial location information. A GCN network is used to perceive row and column information, and finally, the adjacent row and column information of the cell is predicted. i The steps include:

[0048] Step S601: Using the location information obtained from the cell spatial location detection branch network as nodes, calculate the distance to form an adjacency matrix, and construct the graph G = (V, A) of the entire table, where V is the corresponding cell and A is the adjacency matrix of the nodes; for each cell or node V i , with b i and l i b represents its two relational attributes, spatial and logical, respectively. i It includes the cell's center x and y axis position information and its relative width and height, l i Indicates the inferred row and column relationships of adjacent cells;

[0049] Step S602, the spatial features V extracted by the cell spatial location detection branch network are... i s ∈i 256 and multi-scale features F decode The convolutional feature F is obtained by performing RoIAlign operation. cnn The calculation formula is equation (5); the positional features F of the fusion cell spatial location detection branch network. box and F cnn F is obtained by integrating spatial and logical connection relationships. fusion As the input to the subsequent GCN network, the calculation formula is Equation (6);

[0050] F cnn =RoIAlign(F decode V i s (5)

[0051] F fusion =Concat(F box F cnn (6)

[0052] The row and column features of the graph are extracted by designing a row-column GCN convolution combined with edge weights, where the formulas used are equations (7) and (8), where F row Row features and column features F col To obtain the fused feature F in the previous step of GCN network perception. fusionAfter, each cell obtained by performing the Relu activation function is subjected to row and column classification evaluation; the adjacent row and column probability of the predicted cell is obtained from multiple candidate cells, and the adjacent row and column prediction of the cell i is selected as the maximum score, and the starting row End row Starting column End column

[0053] F row = Relu{GCN row (F fusion )} (7)

[0054] F col = Relu{GCN col (F fusion )} (8)

[0055] To achieve the above object, the present application further provides a cell region identification system of a table image, which comprises a memory, a processor and a cell region identification program of a table image stored in the memory, wherein the cell region identification program of the table image is used to execute the steps of the above method when executed by the processor.

[0056] To achieve the above object, the present application further provides a computer readable storage medium, which stores a cell region identification program of a table image, wherein the cell region identification program of the table image is used to execute the steps of the above method when executed by a processor.

[0057] The present application has the following advantages:

[0058] The present application improves the recognition accuracy of a table image, and provides better feature representation and better generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a flowchart of a preferred embodiment of the cell region identification method of the table image of the present application;

[0060] Figure 2 is an overall network model architecture diagram related to the preferred embodiment of the cell region identification method of the table image of the present application;

[0061] Figure 3 is a matrix decomposition axis attention head flow structure diagram;

[0062] Figure 4 is an initial network construction diagram of an original table image;

[0063] Figure 5It is a schematic diagram of the recognition effect of the table image.

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and examples. DETAILED DESCRIPTION

[0065] It should be understood that the specific examples described herein are intended to be illustrative only and are not used to limit the present application.

[0066] Please refer to Figure 1 The present application provides a cell region recognition method for a table image. The preferred embodiment of the cell region recognition method for a table image of the present application comprises the following steps:

[0067] Step S10, input the image data of the document table, set I∈i 3×H×W , wherein H and W represent the height and width of the input image, respectively;

[0068] Step S20, capture the image information of the document table by taking ResNet50 as the backbone network, process the image information of the document table, and obtain multi-level feature information;

[0069] Step S30, fuse the multi-level feature information by means of a feature pyramid network (FPN) to obtain internal information features F inter of the table image;

[0070] Step S40, reorganize the internal information features F inter of the table image based on a low-rank decomposition axis attention mechanism to enhance the connection between the cells;

[0071] Step S50, perform a row, column and cell segmentation task on the table image based on a cell spatial position detection branch network to obtain the spatial position b i of the cell;

[0072] Step S60, fuse the internal information features F inter of the table image with the spatial position information based on a cell logical information branch network to construct a table graph network, perceive the row and column information by using a GCN network, and finally predict the adjacent row and column information l i of the cell.

[0073] Step S70, after fusing the spatial position information and the logical information of the table image, map the reorganized recognition information on the table image.

[0074] Further, in the embodiment, the step S20 includes the following steps of capturing image information of the document table by taking ResNet50 as a backbone network, processing the image information of the document table, and obtaining multi-level feature information.

[0075] The output sizes of f1, f2, f3, and f4 are 4, 8, 16, and 32, respectively, which represent 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image information. These scales correspond to different levels of the pyramid FPN network and are used for detecting targets of different sizes.

[0076] Further, in the embodiment, the step S30 includes the following steps of fusing the multi-level feature information by using the feature pyramid FPN network to obtain internal information features F inter of the table image.

[0077] The feature pyramid FPN network sequentially performs upsampling operations on the bottom feature maps of the network to obtain high-resolution feature maps, and performs downsampling operations on the high-level feature maps to obtain low-resolution feature maps.

[0078] Further, in the embodiment, the step S40 includes the following steps of reorganizing the internal information features F inter of the table image based on a low-rank decomposition axis attention mechanism to enhance the connection between the cells.

[0079] In step S401, the matrix decomposition head is used to reorganize the two-dimensional information of the table captured by the ResNet50 network and the feature pyramid FPN network to enhance the connection between the cells.

[0080] In step S402, the low-rank decomposition axis attention is used to reorganize the global information of the table image.

[0081] Further, in the embodiment, the step of reorganizing the global information of the table image by using the low-rank decomposition axis attention includes the following steps of:

[0082] In step S403, the internal features F inter are input as a matrix with a specification of [B, C, H, W]. During the specification reorganization process, the channel number C of the input tensor is decomposed into S small matrices with a channel number D. Then, each small matrix is expanded in the spatial dimension to form a small tensor with a shape of [B*S, D, N], where N is obtained by multiplying the spatial dimensions H and W.

[0083] Step S404, initializing a base matrix bases with a size of [B*S, D, R], where R represents the rank of the base matrix; initializing a coef matrix, which is equal to the product of the reordering matrix obtained in step S403 after 1D, 2D transformation and multiplication with the base matrix bmm, with a size of [B*S, N, R].

[0084] Step S405, under the constraint of the number of times, continuously performing a local reasoning process to obtain a final base matrix bases and a coefficient matrix coef, where the constraint is that the maximum number of iterations is 7; the local reasoning process is as shown in formulas 1 to 4, where the numerator and denominator tensors correspond to the update rules of the coef and bases matrices, the numerator is the product of the input matrix X and the current base matrix base, and the denominator is the product of the current weight matrix coef and the transposed current base matrix base;

[0085] numerator = BMM(X, base) (1)

[0086] denominator = BMM(coef, BMM(transpose(base), base)) (2)

[0087]

[0088] Step S406, multiplying the base matrix bases and the coefficient matrix coef to obtain a matrix with a size of [B*S, D, N].

[0089] Step S407, reordering the matrix with a size of [B*S, D, N] into a matrix with a size of [B, C, H, W] to obtain the result of matrix decomposition.

[0090] Step S408, transforming into an X = [B, C, W, H] matrix, and performing row-axis convolution on X to obtain a weighted tensor row with a size of [B, C, W, 1] att , X = X x row att .

[0091] Step S409, performing column-axis convolution on X to obtain a weighted tensor col with a size of [B, C, 1, H] att , X = X x col att .

[0092] Step S410, reordering X through 1x1 convolution and outputting, with a size of [B, C, W, H] matrix.

[0093] Further, in the embodiment, the step S50, based on the cell space position detection branch network, performs the row, column and cell segmentation task on the table image to obtain the spatial position b of the cell i The step includes:

[0094] The cell space position detection branch network decodes using a 3x3 convolution kernel, and the input channel is reduced from 1024 to 256.

[0095] The row feature matrix f row ∈i 256×H , the column feature matrix f col ∈i 256×W and the cell feature matrix f cell ∈i 256×H×W are obtained through the row convolution module, the column convolution module and the cell convolution module, wherein the specification of the row feature matrix is [256, image height], the specification of the column feature matrix is [256, image width], and the specification of the cell feature matrix is [256, image height, image width].

[0096] The convolution kernel is sampled to 64, and a 1x1 convolution kernel is used to output the predicted segmentation map y full , wherein The scale specification is [category, image height, image width], and K is the predicted classification category: background, boundary, and cell.

[0097] The mapping information is output as a table segmentation mask image after post-processing in the step S70.

[0098] Further, in the embodiment, the step S60, based on the cell logical information branch network, fuses the internal information features F inter of the table image with the spatial position information to construct a table graph network, perceives the row and column information using a GCN network, and finally predicts the adjacent row and column information l i of the cell.

[0099] In step S601, the position information obtained by the cell space position detection branch network is used as a node to calculate the distance to form an adjacency matrix, and a graph G=(V, A) of the entire table is constructed, wherein V is a corresponding cell, and A is an adjacency matrix of the node; for each cell or node V i , b i and l i represent two relationship attributes of its space and logic, respectively, b i contains the center x-y axis position information and the relative width and height of the cell, and l i represents the adjacent row and column relationship of the cell.

[0100] Step S602, the spatial features V extracted by the cell spatial location detection branch network are... i s ∈i 256 and multi-scale features F decode The convolutional feature F is obtained by performing RoIAlign operation. cnn The calculation formula is equation (5); the positional features F of the fusion cell spatial location detection branch network. box and F cnn F is obtained by integrating spatial and logical connection relationships. fusion As the input to the subsequent GCN network, the calculation formula is Equation (6):

[0101] F cnn =RoIAlign(F decode V i s (5)

[0102] F fusion =Concat(F box F cnn (6)

[0103] The row and column features of the graph are extracted by designing a row-column GCN convolution combined with edge weights, where the formulas used are equations (7) and (8), where F row Row features and column features F col To obtain the fused feature F in the previous step of GCN network perception. fusion Next, the ReLU activation function is used to evaluate the row and column classification of each cell. Among the multiple possible candidates, the probability of the neighboring rows and columns of the predicted cell is calculated. The cell with the highest score is selected for predicting the neighboring rows and columns of cell i, and the row and column prediction begins. End line Start Column End column

[0104] F row =Relu{GCN row (F fusion )} (7)

[0105] F col =Relu{GCN col (F fusion )} (8).

[0106] The following combination Figures 2 to 5 The method for cell region recognition of table images according to the present invention will be further described in detail.

[0107] in, Figure 2It is the overall network model architecture involved in the cell area recognition method of the table image of the present application, Figure 3 It is a matrix decomposition axis attention head flow structure diagram, Figure 4 It is an initial network construction diagram of the original table image, Figure 5 It is a recognition effect diagram of the table image.

[0108] 1, input. The input of the whole model is the image data of the document table, set I∈i 3×H×W Indicates the input table image, wherein H and W represent the height and width of the input image respectively.

[0109] 2, backbone network. The table image has a cell structure structure complex and contains a lot of shallow information, so the ResNet50 is taken as the backbone network to capture the image information. The output size is 4, 8, 16, 32 f1, f2, f3, f4 (corresponding to the feature information of the hierarchical feature), which is 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image information respectively. These scales correspond to different levels of pyramid, which are used to detect targets of different sizes. The feature pyramid (FPN) obtains high-resolution feature maps from the bottom feature maps of the network (usually the output of the backbone network) through upsampling (upshift) operation, while the high-level feature maps obtain low-resolution feature maps through downsampling (downshift) operation.

[0110] 3, four-layer pyramid. The internal information features F of the table image are obtained by fusing the multi-level feature information with the help of the feature pyramid FPN inter .

[0111] 4, matrix decomposition head. After obtaining sufficient image features through the backbone network, the matrix tap is used to further process the information. The table two-dimensional information captured by the backbone network and FPN in the last step is reorganized to enhance the connection between the cells.

[0112] The present application uses a low-rank decomposition axis attention pair to reorganize the global information, which has more effective performance and computational overhead than the single attention mechanism. As shown in Figure 2 In the low-rank decomposition axis attention algorithm,

[0113] (1) input internal feature F inter It is a matrix with the specification [B, C, H, W]. In the specification reorganization process, we decompose the channel number C of the input tensor into S small matrices with channel number D, and then expand each small matrix in the spatial dimension to form a small tensor with shape [B*S, D, N]. Here, N is obtained by multiplying the spatial dimensions H and W.

[0114] (2) the maximum iteration number is constrained to be 7.

[0115] (3) Initialize the base matrix bases with size [B*S, D, R], where R represents the rank of the base matrix. Initialize the coef matrix, which equals the product of the reorganized matrix obtained in the first step after 1D, 2D transformation and multiplication with the base matrix bmm, with size [B*S, N, R]

[0116] (4) Under the constraint of the number of times, continuously perform the local reasoning process to obtain the final base matrix bases and the coefficient matrix coef. The local reasoning process is as shown in Formulas 1 to 4, where the numerator and denominator tensors correspond to the update rules of the coef and bases matrices. The numerator is the product of the input matrix X and the current base matrix base, and the denominator is the product of the current weight matrix coef and the transpose of the current base matrix base. The non-negativity constraint is forced by setting negative values in the matrix to zero, ensuring that these matrices remain non-negative during all iteration processes, thereby providing a more meaningful interpretation.

[0117] (5) Multiply the base matrix bases and the coefficient matrix coef to obtain a matrix with size [B*S, D, N].

[0118] (6) Reorganize the matrix with size [B*S, D, N] into a matrix with size [B, C, H, W] to obtain the result of matrix decomposition.

[0119] (7) Transform into an X=[B, C, W, H] matrix, and perform row-axis convolution on X to obtain a weighted tensor row with size [B, C, W, 1] att , X=Xxrow att .

[0120] (8) Perform column-axis convolution on X to obtain a weighted tensor col with size [B, C, 1, H] att , X=Xxcol att .

[0121] (9) Reorganize X after 1x1 convolution and output, with size [B, C, W, H] matrix.

[0122] (10) The following parameters are involved:

[0123] B: represents the number of samples in a batch.

[0124] C: represents the number of channels or features of the input image.

[0125] H: represents the height of the input image.

[0126] W: represents the width of the input image.

[0127] S: represents the number of channels or features after decomposition, i.e., the number of channels or features of each matrix after decomposition. The initial value is 1.

[0128] D: represents the number of elements in each small matrix, i.e., the depth. The initial value is 512.

[0129] N: represents the number of matrices, i.e., the number of matrices after decomposition.

[0130] R: represents the rank of the matrix, and the initial value of the rank of the basis matrix is 64.

[0131] @: bmm operation, i.e., batch matrix multiplication.

[0132] 5, cell space information branch, reorganizes the two-dimensional information existing in the table, enhances the representation of the relationship between cells, and decodes to send to the cell space position detection branch for row, column, and cell segmentation tasks to obtain the spatial position of the cell b i .

[0133] After the matrix decomposition reorganizes the multi-layer image information F inter captured by the backbone network, the spatial position detection first uses a 3x3 convolution kernel for decoding, and the input channel is reduced from 1024 to 256. Considering that the table data is arranged by rows and columns, the row convolution module, the column convolution module, and the cell convolution module are used to obtain the row feature matrix (f row ∈i 256×H , specification: [256, image height]), column feature matrix (f col ∈i 256×W , specification: [256, image width]), and cell feature matrix (f cell ∈i 256×H×W , specification: [256, image height, image width]). Further sampling to 64 using a convolution kernel, and finally using a 1x1 convolution kernel output to obtain the predicted segmentation map where the scale specification is [category, image height, image width], and K is the predicted classification category (“background”, “boundary”, “cell”). The mapping information is output as a segmented mask image of the table after post-processing in the seventh step.

[0134] 6, cell logical information branch, fuses F inter with spatial position information in the logical position prediction branch to construct a graph network of the table, uses GCN to perceive row and column information, and finally predicts the adjacency row and column information of the cell l i .

[0135] (1) Using the location information obtained from cell space detection as nodes, calculate the distance to form an adjacency matrix, and construct the graph G = (V, A) of the entire table, where V is the corresponding cell and A is the adjacency matrix of the nodes. Therefore, for each cell or node V i , with b i and l i These represent its spatial and logical relational attributes, respectively. i It includes the cell's center x and y axis position information, as well as its relative width and height. i This indicates the inferred row and column relationships of adjacent cells.

[0136] (2) Spatial features V extracted from the spatial location detection branch i s ∈i 256 and multi-scale features F decode The convolutional feature F is obtained by performing RoIAlign operation. cnn As shown in formula (5). Finally, the positional features F of the spatial detection branch are fused. box and F cnn F is obtained by integrating spatial and logical connection relationships. fusion As input to the subsequent GCN, as in formula (6).

[0137] (3) The initialized graph is a directed complete graph, such as Figure 3 As shown in the diagram, row and column features of the graph are extracted by designing row-column GCN convolutions combined with edge weights, as shown in Equations 7 and 8. row Row features and column features F col The GCN module perceives the fused feature F obtained in the previous step. fusion Next, the ReLU activation function is used to evaluate the row and column classification of each cell. For multiple candidate cells, the probabilities of the neighboring rows and columns of the predicted cell are calculated, and the cell with the highest score is selected for predicting the neighboring rows and columns of cell i. (Start line), (End line) (Start column) (End of column).

[0138] 7. Post-processing: After fusing spatial and logical information, the reconstructed recognition information is mapped onto the tabular image. This is displayed as a mask layer. Regarding the recognition results, there are two scenarios:

[0139] (1) Light-colored background indicates consistent testing.

[0140] (2) Dark background indicates an error.

[0141] The main innovations of the cell region recognition method for table images in this invention are as follows:

[0142] 1. A dual-branch network model is designed, which takes ResNet+FPN as the backbone network, detects the cell spatial position information and predicts the cell adjacency logic, and provides the cell spatial position information and the possible adjacency information of the cell.

[0143] 2. The low-rank decomposition axis attention is designed according to the characteristics of the table image, that is, the low-rank information is first decomposed and reorganized to reduce the calculation amount of the attention mechanism, and then the row-column axis attention mechanism is used to capture the key boundary information. In the table image structure recognition, the low-rank decomposition axis attention has better performance and lower calculation overhead.

[0144] 3. The graph network is designed to build the connection between cells. Considering the internal relationship between the table and the graph network in the cell logic prediction network, the graph convolution network module is introduced. The network integrates the cell position information and the cell logic information, so that the network can more accurately infer the connection between cells, thereby improving the model recognition accuracy.

[0145] 4. The pre-trained multi-modal large model is combined to realize the alignment of the image mode to the sequence text mode, and the general language and image representation is learned through learning a large-scale data set and a predefined task, which can provide better feature representation and better generalization ability.

[0146] The basic idea of the cell region recognition method of the table image of the present application is as follows:

[0147] 1. How to design the backbone network of the table image task to maximize the capture of table image information, so as to ensure the detection accuracy of the table items at a suitable granularity.

[0148] Basic idea: increase axis attention to better capture row and column information; design constraints to apply effective constraints to constrain granularity.

[0149] 2. How to design the network model architecture of the overall model to effectively capture the entire spatial information and logical information and utilize them.

[0150] Basic idea: ResNet-50 and FPN are used as the backbone network to extract multi-scale feature representations from the input table image. The cell spatial position detection and logical position prediction are jointly performed by two multi-task head branches. The semantic segmentation is used to obtain the cell spatial information, and the GCN is used to learn the logical information.

[0151] 3. How to recover and reconstruct the structure of the table items after obtaining the region detection information of the table image.

[0152] Basic idea: the identification of structural information is the premise of the entire task, and the structural information is the embodiment of the position information and the subordinate information of each table item. The entire task is regarded as an ordered node classification, and a graph convolution network of the table item is constructed by combining the graph neural network to restore and reconstruct the cell.

[0153] 4. How to introduce multi-modal information to realize the generation of corresponding label sequences of document images

[0154] Basic idea: the document image sequence generation model algorithm combined with the pre-training large model can be applied to the automatic generation and automatic processing of document images, including automatic text summary, text generation, image generation, etc. The document image sequence generation model algorithm can extract the feature representation of text and image by using the pre-training large model, and then generate new text sequences combined with the generation model algorithm. Specifically, the generation model can use algorithms such as recurrent neural network (RNN), variational autoencoder (VAE), and Transform, and fine-tune the training on the basis of the pre-training large model to realize the automatic generation of label sequences of document images.

[0155] 5. How to solve the problem of poor logical relationship capture performance and low overall efficiency of GCN for large tables.

[0156] Basic idea: try to design different GCN module structures, such as Muti-layer GCN and R-GCN, to increase its perception performance. Improve the loss of large table information by fusing spatial information and logical information through algorithm.

[0157] 6. How to solve the problem of poor detection effect for widely used but complex tables (weak generalization performance).

[0158] Basic idea: increase the proportion of complex data in the training data set and increase the verification data set. In the preprocessing process, the image data is processed by distortion, noise increase, ripple and color difference.

[0159] The beneficial effects of the cell region identification method for table images according to the present application are as follows:

[0160] The present application improves the recognition accuracy of table images, and can provide better feature representation and better generalization ability.

[0161] To achieve the above purpose, the present application also proposes a cell region identification system for table images, which comprises a memory, a processor and a cell region identification program for table images stored in the memory. The cell region identification program for table images is run by the processor to execute the steps of the method as described above, which will not be repeated here.

[0162] To achieve the above object, the application further provides a computer readable storage medium, which stores a cell region identification program of a table image, and the cell region identification program of the table image performs the steps of the method when run by a processor, which will not be repeated here.

[0163] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or process transformation made by using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for recognizing cell regions in a table image, characterized in that, Includes the following steps: Step S10: Input the image data of the document table, let I∈i 3×H×W This represents the input table image, where H and W represent the height and width of the input image, respectively. Step S20: Capture image information of document tables using ResNet50 as the backbone network, and process the image information of the document tables to obtain multi-level feature information; Step S30: Using the Feature Pyramid Network (FPN), the feature information from the multiple levels is fused to obtain the internal information feature F of the table image. inter ; Step S40: Based on the low-rank decomposition axis attention mechanism, the internal information features F of the table image are analyzed. inter Reorganize to enhance the connections between representation cells; Step S50: Based on the cell spatial location detection branch network, the table image is segmented into rows, columns, and cells to obtain the spatial location b of the cells. i ; Step S60: Based on the cell logical information branch network, fuse the internal information features F of the table image. inter A graph network is constructed for the table based on spatial location information. A GCN network is used to perceive row and column information, and finally, the adjacent row and column information of the cell is predicted. i ; Step S70: After fusing the spatial location information and logical information of the table image, the reconstructed recognition information is mapped onto the table image. In step S40, the internal information features F of the table image are analyzed based on the low-rank decomposition axis attention mechanism. inter The steps to reorganize and enhance the relationships between representation cells include: Step S401: The matrix factorization head is used to reorganize the two-dimensional information of the table captured by the ResNet50 network and the Feature Pyramid Network (FPN) to enhance the connection between the representation cells. Step S402: Use low-rank decomposition axis attention pairs to reorganize the global information of the table image; The step of reconstructing global information of a table image using low-rank decomposition axis attention pairs includes: Step S403, input internal feature F inter Given a matrix of size [B, C, H, W], during size renormalization, the input tensor's channel count C is decomposed into S smaller matrices with D channels. Each smaller matrix is ​​then expanded in spatial dimensions to form a small tensor of shape [B*S, D, N]. Here, B represents the number of samples in the batch, C represents the number of channels or features of the input image, S represents the number of channels or features after decomposition, D represents the number of elements in each smaller matrix (i.e., depth), and N represents the number of matrices (i.e., the number of matrices after decomposition), which is obtained by multiplying the spatial dimensions H and W. Step S404: Initialize the base matrix bases with size [B*S, D, R], where R represents the rank of the base matrix; initialize the coef matrix, which is equal to the product of the renormal matrix obtained in step S403 after 1D and 2D transformations and the bases matrix bmm, with size [B*S, N, R]. Step S405: Under the constraint number, the local reasoning process is continuously performed to obtain the final base matrix bases and coefficient matrix coef, where the maximum number of constraint iterations is 7; the local reasoning process is as shown in formulas (1) to (4), where the numerator and denominator tensors correspond to the update rules of the coef and bases matrices, the numerator is the product of the input matrix X and the current base matrix base, and the denominator is the product of the current weight matrix coef and the transposed current base matrix base; numerator = BMM(X, base) (1) denominator=BMM(coef,BMM(transpose(base),base)) (2) Step S406: Multiply the base matrix bases and the coefficient matrix coef to obtain a matrix of size [B*S, D, N]. Step S407: Rearrange the matrix of size [B*S, D, N] into a matrix of size [B, C, H, W] to obtain the matrix decomposition result; Step S408: Transform the matrix into X = [B, C, W, H], and perform row convolution on X to obtain a weighted tensor row of size [B, C, W, 1]. att X = X × row att ; Step S409: Perform column-axis convolution on X to obtain a weighted tensor col of size [B, C, 1, H]. att X = X × col att ; Step S410: Reshape X through 1×1 convolution and output a matrix of size [B, C, W, H]. Step S50, which involves segmenting the table image into rows, columns, and cells based on the cell spatial location detection branch network to obtain the spatial location bi of the cells, includes: The cell spatial location detection branch network uses 3×3 convolutional kernels for decoding, reducing the number of input channels from 1024 to 256. The row feature matrix f is obtained through row convolution module, column convolution module, and cell convolution module. row ∈i 256×H Column characteristic matrix f col ∈i 256×W and cell feature matrix f cell ∈i 256×H×W The row feature matrix has a size of [256, image height], the column feature matrix has a size of [256, image width], and the cell feature matrix has a size of [256, image height, image width]. The convolution kernel samples to 64, and the predicted segmentation map is obtained by using a 1×1 convolution kernel output. in, The scale is obtained as [category, image height, image width], where K is the predicted classification category: background, border, cell; After post-processing in step S70, the mapping information is output as a segmented mask image of a table. In step S60, the internal information features F of the table image are fused based on the cell logical information branch network. inter A graph network is constructed for the table based on spatial location information. A GCN network is used to perceive row and column information, and finally, the adjacent row and column information of the cell is predicted. i The steps include: Step S601: Using the location information obtained from the cell spatial location detection branch network as nodes, calculate the distance to form an adjacency matrix, and construct the graph G = (V, A) of the entire table, where V is the corresponding cell and A is the adjacency matrix of the nodes; for each cell or node V i , with b i and l i b represents its two relational attributes, spatial and logical, respectively. i It includes the cell's center x and y axis position information and its relative width and height, l i Indicates the inferred row and column relationships of adjacent cells; Step S602, the spatial features V extracted by the cell spatial location detection branch network are... i s ∈i 256 and multi-scale features F decode The convolutional feature F is obtained by performing RoIAlign operation. cnn The calculation formula is equation (5); the positional features F of the fusion cell spatial location detection branch network. box and F cnn F is obtained by integrating spatial and logical connection relationships. fusion As the input to the subsequent GCN network, the calculation formula is Equation (6); F cnn =RoIAlign(F decode ,V i s ) (5) F fusion =Concat(F box ,F cnn ) (6) The row and column features of the graph are extracted by designing a row-column GCN convolution combined with edge weights, where the formulas used are equations (7) and (8), where F row Row features and column features F col To obtain the fused feature F in the previous step of GCN network perception. fusion Next, the ReLU activation function is used to evaluate the row and column classification of each cell. Among the multiple possible candidates, the probability of the neighboring rows and columns of the predicted cell is calculated. The cell with the highest score is selected for predicting the neighboring rows and columns of cell i, and the row and column prediction begins. End line Start Column End column F row =Relu{GCN row (F fusion )} (7) F col =Relu{GCN col (F fusion )} (8).

2. The method for recognizing cell regions of a table image according to claim 1, characterized in that, Step S20, which involves capturing image information of document tables using ResNet50 as the backbone network and processing the image information of the document tables to obtain multi-level feature information, includes the following steps: The output sizes f1, f2, f3, and f4 are 4, 8, 16, and 32, respectively, representing 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image information. These scales correspond to different levels of the pyramid FPN network and are used to detect targets of different sizes.

3. The method for recognizing cell regions of a table image according to claim 2, characterized in that, In step S30, the feature information from the multiple levels is fused using a Feature Pyramid Network (FPN) to obtain the internal information feature F of the table image. inter The steps include: The Feature Pyramid Network (FPN) sequentially upsamples the lower-level feature maps to obtain high-resolution feature maps, and downsamples the higher-level feature maps to obtain low-resolution feature maps.

4. A cell region recognition system for a table image, characterized in that, The system includes a memory, a processor, and a cell region recognition program for a table image stored in the memory, the cell region recognition program for the table image being executed by the processor to perform the steps of the method as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a cell region recognition program for a table image, which, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 3.

Images