Table image recognition method, device, equipment and medium

Abstract

The present application provides a table image recognition method, device, equipment and medium, comprising: performing table line pixel marking on a to-be-recognized table image to obtain a semantic segmentation binary image; sequentially performing image correction, table line extraction and table line position determination on the semantic segmentation binary image to determine the position coordinates of the table line; determining the start and end row and column of a cell based on the position coordinates of the table line; performing single-character position segmentation on the to-be-recognized table image to determine character position information, which represents the position information of each character; and based on the start and end row and column and the character position information, aggregating the cells and the corresponding text content to generate a table file. The present application is used to solve the defect that the table detection and positioning are not accurate enough in the prior art, and to improve the accuracy of table detection and positioning.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, device, and medium for recognizing table images. Background Technology

[0002] Tables are a common way to display formatted data and are widely used in scientific research, data analysis, and other fields. With the continuous improvement of information technology, locating and retrieving desired content from non-editable paper tables has become a pressing issue that needs to be addressed.

[0003] Among the methods for making tables editable, the most widely used approach is table detection + table cell detection + table structure recognition. Table detection identifies the area containing the table from images or other files; table cell detection identifies all cells appearing in the table, including merged cells; table structure recognition parses the table area, extracts the text content and structural information, and obtains the distribution of rows and columns and the logical structure between cells, also known as table document reconstruction.

[0004] As a general data organization structure, tables lack specific standards, resulting in inconsistent styles and types, which greatly increases the difficulty of recognition tasks. Furthermore, interference from various factors such as different fill colors, different text types in each column, and different image types (screenshots, PDFs, photos) makes table recognition a persistent research challenge in the field of document recognition.

[0005] The first type of recognition method is a traditional image processing method, which uses image morphological transformation, texture extraction, edge detection, and other techniques to extract table lines. Finally, the information of rows, columns, and cells is derived from the table lines. However, this method relies too heavily on traditional image processing algorithms, which require setting thresholds. This means that a set of parameters may be effective for one batch of data but ineffective for another, resulting in poor robustness. Furthermore, for tables without visible lines, table detection and localization are not accurate enough.

[0006] The second type of recognition method uses OCR technology to deduce row, column, and cell information based on the position information of the OCR text detection box, thereby generating a spreadsheet. However, this method is highly dependent on the OCR detection results and manually designed rules, and the table detection and positioning are not precise enough.

[0007] The third type of method is end-to-end neural network learning, which uses image-to-text sequence technology to convert table images into some kind of structured language such as HTML tags. However, this method requires that the sequence length cannot be too long, so it performs poorly for tables with dense cells, and the table detection and localization are not accurate enough.

[0008] Therefore, there is an urgent need for a table image recognition method that can improve the accuracy of table detection and localization. Summary of the Invention

[0009] This invention provides a table image recognition method, apparatus, device, and medium to address the shortcomings of inaccurate table detection and positioning in existing technologies, thereby improving the accuracy of table detection and positioning.

[0010] This invention provides a table image recognition method, comprising:

[0011] The table image to be recognized is marked with table line pixels to obtain a semantic segmentation binarized image;

[0012] The semantic segmentation binarized image is sequentially subjected to image correction, table line extraction, and table line position determination to determine the position coordinates of the table lines;

[0013] Based on the position coordinates of the table lines, determine the start and end rows and columns of the cells;

[0014] The table image to be identified is segmented into single characters to determine the character position information, which represents the position information of each character.

[0015] Based on the start and end rows and columns and the character position information, the cells and their corresponding text content are aggregated to generate a table file.

[0016] According to the table image recognition method proposed in this invention, table lines are extracted and their positions are determined from the semantically segmented binary image. The determination of the table line position coordinates includes:

[0017] Based on the semantic segmentation binarized graph, the connected regions in the semantic segmentation binarized graph are extracted;

[0018] Fit the bounding lines of the connected regions to determine the table lines to be identified;

[0019] The position coordinates of the table line are determined based on the smallest bounding rectangle corresponding to the table line to be identified.

[0020] According to the table image recognition method proposed in this invention, the table image to be recognized is segmented into single-character positions to determine character position information. The character position information represents the position information of each character, including:

[0021] The image of the table to be recognized is input into a convolutional recurrent neural network to obtain a probability matrix of the sequence vector length and the number of characters output by the convolutional recurrent neural network.

[0022] The probability matrix is ​​used to determine the character segmentation position and obtain the character position information. If the character is a Chinese character, the index of the first empty sequence vector is determined as the starting index of the Chinese character, and the starting index of the Chinese character is used to determine the starting coordinates of the Chinese character. If the character is a character other than a Chinese character, the index of the second empty sequence vector is determined as the starting index of the other character, and the starting index of the other character is used to determine the starting coordinates of the other character. The starting coordinates are used to determine the character segmentation position.

[0023] According to the table image recognition method proposed in this invention, based on the start and end row and column information and the character position information, the method aggregates the cells with the corresponding text content to generate a table file, including:

[0024] Input the start and end rows and columns and the character position information into the aggregation model, aggregate the character boxes and cell boxes, and generate a table file;

[0025] The aggregation model is used to match text content with corresponding cells, and the algorithms applied by the aggregation model include the central dogma and the maximum IOU rule.

[0026] According to the table image recognition method proposed in this invention, image correction is performed on the semantic segmentation binarized image, including:

[0027] The semantic segmentation binarized image is input into the rotating target detection model of the corner regression algorithm to determine the table region and table corner of the semantic segmentation binarized image;

[0028] The table area and the table corner points are used to perform image correction on the semantic segmentation binarized image.

[0029] According to the table image recognition method proposed in this invention, the rotating target detection model is an improved network based on the YOLOv5 network. The improved network structure is an angle category branch and a key point regression branch added to the structure of the YOLOv5 network.

[0030] The present invention also provides a table image recognition device, comprising:

[0031] The semantic segmentation module is used to mark the table line pixels in the table image to be recognized, and obtain a semantic segmentation binarized image.

[0032] The table line position determination module is used to sequentially perform image correction, table line extraction, and table line position determination on the semantic segmentation binarized image, and determine the position coordinates of the table lines.

[0033] The cell determination module is used to determine the start and end rows and columns of a cell based on the position coordinates of the table lines.

[0034] The character segmentation module is used to segment the table image to be recognized into single characters and determine the character position information, wherein the character position information represents the position information of each character.

[0035] The aggregation module is used to aggregate the cell and its corresponding text content based on the start and end rows and the character position information, and generate a table file.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the table image recognition method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the table image recognition method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the table image recognition method as described above.

[0039] The table image recognition method, apparatus, device, and medium provided by this invention obtain a semantic segmentation binarized image by semantic segmentation of the image to be recognized. Based on the semantic segmentation binarized image, image correction, table line extraction, and cell position determination are completed. Then, table structure recognition and text recognition within the table are performed. Finally, the text content and cells are aggregated to obtain a table file. By performing table structure recognition and text position recognition separately after semantic segmentation, the accuracy of table detection and positioning is improved. On the other hand, it realizes the efficient conversion of table images into table files, effectively improving the user's office efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating the table image recognition method provided by the present invention;

[0042] Figure 2 This is the second flowchart illustrating the table image recognition method provided by the present invention;

[0043] Figure 3This is the third flowchart of the table image recognition method provided by the present invention;

[0044] Figure 4 This is the fourth flowchart of the table image recognition method provided by the present invention;

[0045] Figure 5 This is the fifth flowchart illustrating the table image recognition method provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following is combined with Figures 1-5 The table image recognition method of the present invention is described.

[0049] Please refer to Figure 1 and Figure 2 The table image recognition method proposed in this invention includes:

[0050] Step 10: Mark the table lines in the image to be recognized with pixels to obtain a semantic segmentation binarized image;

[0051] In this embodiment of the application, a table image to be recognized is obtained for table image recognition. The table image to be recognized supports common image formats such as PNG, JPG, and JPEG, as well as PDF text.

[0052] It should be noted that the innovative table detection model and semantic segmentation binarized image are established. The table detection is completed through the table detection model. Specifically, the table image to be identified is input into the table detection model, which marks the table lines of the table image with pixels to complete the table detection and obtain the semantic segmentation binarized image, which includes the pixel markings of the table lines.

[0053] Step 20: Perform image correction, table line extraction, and table line position determination on the semantic segmentation binarized image in sequence to determine the position coordinates of the table lines;

[0054] A semantic segmentation binarized image is obtained, an innovative semantic segmentation model is established, and geometric analysis is performed on the segmentation result, i.e., the semantic segmentation binarized image, to achieve image correction, table line extraction, and cell position coordinate determination. The geometric analysis operations include image correction, table line extraction, and table line position determination. Specifically, based on the pixel markers of the table lines in the semantic segmentation binarized image, the shape of the table lines is determined, and then the table lines are corrected according to their shape. Since tables in table images are usually tilted or trapezoidal, trapezoidal table images need to be corrected before table line extraction and position determination to improve recognition performance. Then, based on the corrected table line pixel markers, table line extraction and position determination are performed to determine the position coordinates of the table lines, ultimately extracting the table line positions. The position coordinates of the table lines are used to characterize the table line positions.

[0055] To achieve efficient and accurate semantic segmentation, the innovative semantic segmentation model can adopt the innovative ErfNet real-time semantic segmentation model. For the backbone design, referencing MobileNet, depthwise + pointwise convolutions are used instead of conventional convolutions. Table lines are elongated objects, and a larger receptive field in the horizontal and vertical directions will bring more benefits. Therefore, 5x1 and 1x5 convolutional kernel shapes are selected, which, in practice, achieve better performance than the commonly used 3x3. Furthermore, because there are non-exclusive labels in the table line segmentation scenario, sigmoid is used instead of softmax to achieve multi-label classification and obtain the semantic segmentation binary image. As the sigmoid function approaches positive or negative infinity, it becomes smoother. Because its output is between 0 and 1, the sigmoid function can be used in multi-label classification.

[0056] Given the imbalance in the number of pixels across different classes, the loss functions used are weighted cross-entropy, DiceLoss, and FocalLoss, with weights of 1.0, 3.0, and 5.0, respectively. To improve model performance and accelerate convergence, the learning rate of the decoding head component is set to be 10 times that of the backbone network. Additionally, Online Hard Example Mining (OHEM) is implemented, using only pixels with confidence scores below 0.7 for training. To enhance the model's generalization ability, the training data undergoes random flipping, rotation, photometric distortion, and geometric distortion transformations.

[0057] Step 30: Determine the start and end rows and columns of the cell based on the position coordinates of the table lines;

[0058] A TSRA table structure inference algorithm model is established to identify table structures. The coordinates of the table lines are input into the TSRA table structure inference algorithm model. Based on the table line positions, the model identifies the table structure, deriving the start and end rows and columns of cells from the table lines, thus completing the table structure identification and obtaining the start and end rows and columns of each cell. These start and end rows and columns are used to represent the table structure.

[0059] Existing table structure reasoning algorithms are prone to errors when dealing with tables with merged cells. To address this, we propose an efficient algorithm that derives the start and end rows and columns of cells from table lines, thereby parsing the table structure.

[0060] Table reconstruction based on cell position information begins by deriving the start and end columns of each cell. The start and end x-coordinates (xmin, xmax) of the top-left and bottom-right corners of all cells are extracted and then sorted in ascending order. A mapping dictionary `edgesMap{'actual x-value':'theoretical x-value'}` is created. The purpose is that in real-world samples, two adjacent cells may have different xmin or xmax values, but theoretically they should be the same. Therefore, a threshold is set: if the difference between two x values ​​is greater than this threshold, they are considered not to be the same theoretical value; otherwise, they are considered to be the same theoretical value. A list storing a series of theoretical values ​​is obtained from the mapping dictionary. This list is sorted in ascending order and processed into the following data structure: `edgesMapIndex{theoretical x0:0, theoretical x1:0, theoretical x2:1, theoretical x3:2…}`. Finally, all cells are traversed, and the start and end x-coordinates of their top-left and bottom-right corners are retrieved. First, the theoretical value is obtained from `edgesMap`, and then the start and end columns are obtained from `edgesMapIndex`. Similarly, the start and end row information of each cell can be derived. This table structure reasoning algorithm is applicable to various scenarios involving merging cells across rows and columns, and it is fast and has good generalization ability.

[0061] Step 40: Perform single-character position segmentation on the table image to be identified to determine character position information, wherein the character position information represents the position information of each character;

[0062] A text position recognition model is established to segment the input table image into individual characters, determine their positions, and perform text recognition and position determination. This model is an improved version of the original text recognition model, capable of recognizing both the text itself and its location. Specifically, the table image to be recognized is input into the text position recognition model to determine the character position information.

[0063] Step 50: Based on the start and end rows and columns and the character position information, aggregate the cells and their corresponding text content to generate a table file.

[0064] An aggregation model is established to match text content with cells. After recognizing the table structure and character position information, the start and end rows and columns of the cells corresponding to the table structure, as well as the character position information, are input into the aggregation model. The aggregation model aggregates the text content with the cells based on the start and end rows and columns and the character position information, thereby generating a table file. The above steps achieve the conversion from a table image to a table file, also known as table document reconstruction. The format of the generated table file can include an Excel spreadsheet, a Word document, or any other format; the generated table file is not limited in this embodiment.

[0065] The table image recognition method provided by this invention obtains a semantic segmentation binarized image by semantic segmentation of the image to be recognized. Based on the semantic segmentation binarized image, image correction, table line extraction, and cell position determination are completed. Then, table structure recognition and text recognition within the table are performed. Finally, the text content and cells are aggregated to obtain a table file. By performing table structure recognition and text position recognition separately after semantic segmentation, the accuracy of table detection and positioning is improved. On the other hand, it realizes the efficient conversion of table images into table files, effectively improving the user's office efficiency.

[0066] In one embodiment, please refer to Figure 3 Step 20 involves extracting table lines and determining their positions on the semantic segmentation binarized image, including determining the coordinates of the table lines.

[0067] Step 201: Based on the semantic segmentation binarized graph, extract the connected regions in the semantic segmentation binarized graph;

[0068] Step 202: Fit the bounding box of the connected region to determine the table line to be identified;

[0069] Step 203: Determine the position coordinates of the table line based on the smallest bounding rectangle corresponding to the table line to be identified.

[0070] In this embodiment, geometric analysis is performed based on the corrected semantic segmentation binarized image. The specific steps are as follows: first, the connected regions are extracted; then, the connected regions are fitted to form bounding boxes; then, the minimum bounding rectangle is calculated; and finally, the position coordinates of all cells are obtained.

[0071] In this embodiment, since the table line pixels in the semantic segmentation binarized image are not necessarily continuous, the position coordinates of the table lines of all cells are determined by calculating the minimum bounding rectangle of the connected regions, thereby improving the accuracy of the position calculation of the table lines of the cells and further improving the accuracy of table recognition and positioning of the table image.

[0072] In one embodiment, step 40 involves segmenting the table image to be recognized into single-character positions to determine character position information, including:

[0073] The image of the table to be recognized is input into a convolutional recurrent neural network to obtain a probability matrix of the sequence vector length and the number of characters output by the convolutional recurrent neural network.

[0074] The probability matrix is ​​used to determine the character segmentation position and obtain the character position information. If the character is a Chinese character, the index of the first empty sequence vector is determined as the starting index of the Chinese character, and the starting index of the Chinese character is used to determine the starting coordinates of the Chinese character. If the character is a character other than a Chinese character, the index of the second empty sequence vector is determined as the starting index of the other character, and the starting index of the other character is used to determine the starting coordinates of the other character. The starting coordinates are used to determine the character segmentation position.

[0075] Existing OCR models lack character-level positional information. Current methods include two approaches: one is projection segmentation, which analyzes the pixel distribution histogram of the binarized image to find the boundary points between adjacent characters for segmentation; the other is connected component segmentation, which first performs opening operations on the image after binarization to remove minor noise, then marks the connected components in the image connected in an eight-neighborhood manner, calculates the bounding box features of the connected component regions, including the coordinates of the top left corner and the length and width, and segments characters based on these data.

[0076] The character segmentation steps using the CTC decoding strategy are as follows: The convolutional recurrent neural network outputs a probability matrix of sequence vector length × number of characters, where the sequence vectors are stride by a certain stride, corresponding to the original image. The CTC stage can obtain the index of the non-empty sequence vector. For Chinese characters, the index of the first empty sequence vector can be used as the starting index of the Chinese character, and the index of the second empty sequence vector is used for other characters. Then, the starting coordinates of other characters can be inferred.

[0077] The first empty sequence vector is related to Chinese characters. It consists of the first few empty sequences of the sequence vector corresponding to each Chinese character, used to adjust the positioning of the Chinese character. The second empty sequence vector is related to other characters. It consists of the first few empty sequences of the sequence vector corresponding to each other character, used to adjust the positioning of the other characters. Determining the first and second empty sequence vectors may include determining the length and type of the sequence vectors. The first and second empty sequence vectors are determined based on the length and type of the sequence vectors. If the length of the sequence vector exceeds a set threshold (usually 1), the empty sequence vector is incremented by 0.5; otherwise, it remains unchanged. If the type of the sequence vector is a Chinese sequence, the empty sequence vector is incremented by 1. For example, if the length of the sequence vector for a Chinese character exceeds 1, the first empty sequence vector corresponding to that Chinese character is the first 1.5 empty sequence vectors; if the length of the sequence vector for other characters exceeds 1, the second empty sequence vector can be the first 0.5 empty sequence vectors.

[0078] In this embodiment, different indexing strategies are used for different character types to make the character segmentation position more accurate, improve the accuracy of character segmentation, and thus further improve the accuracy of character position recognition.

[0079] In addition, the CTC decoding part is implemented using NumPy arrays, which further speeds up the CTC decoding process.

[0080] In one embodiment, please refer to Figure 4 Step 50: Based on the start and end rows and columns and the character position information, aggregate the cells and their corresponding text content to generate a table file, including:

[0081] Step 51: Determine the aggregation model of text content and cells, wherein the aggregation model is used to match text content with corresponding cells;

[0082] Step 52: Input the start and end rows and columns and the character position information into the aggregation model, aggregate the character boxes and cell boxes, and generate a table file;

[0083] The algorithms used in the aggregation model include the centrality rule and the maximum IOU rule.

[0084] An aggregation model is established to match text content with cells. After recognizing the table structure and character position information, the start and end rows and columns of the cells corresponding to the table structure, as well as the character position information, are input into the aggregation model. The aggregation model aggregates the text content with the cells based on the start and end rows and columns and the character position information, thereby generating a table file. Through the above steps, the conversion from a table image to a table file is achieved, also known as table document reconstruction.

[0085] In this embodiment, the text in each cell can be obtained by aggregating the character box and the cell box using the central dogma and the maximum IOU rule.

[0086] In one embodiment, please refer to Figure 5 Step 20, performing image correction on the semantic segmentation binarized image, including:

[0087] Step 211: Determine the rotating target detection model for the corner regression algorithm;

[0088] Step 212: Input the semantic segmentation binarized image into the rotating target detection model to determine the table regions and table corners of the semantic segmentation binarized image;

[0089] The table area and the table corner points are used to perform image correction on the semantic segmentation binarized image.

[0090] In this embodiment, a rotating target detection model with a corner regression algorithm is proposed. This rotating target detection model with a corner regression algorithm can detect the table area and the four corners of the table at the same time, thereby achieving image correction through perspective transformation.

[0091] In one embodiment, the rotating target detection model is an improved network based on the YOLOv5 network, wherein the improved network structure is an angle category branch and a key point regression branch added to the structure of the YOLOv5 network.

[0092] Table detection, or detecting the outer bounding box of a table, is necessary because multiple tables may exist within the same image. In real-world scenarios, images may be tilted or distorted, making non-rotational object detection algorithms less suitable (as they use horizontal quadrilateral boxes to define the target's position). Furthermore, existing table detection algorithms lack a corner regression module, failing to accurately locate table corner positions and thus hindering perspective transformation and image correction.

[0093] Employing a rotating target detection model with corner regression algorithm facilitates accurate table localization, providing more accurate initial results for subsequent advanced tasks such as recognition and analysis. The model uses an improved rotating target detection algorithm based on YOLOv5. This is achieved by adding angle category branches and keypoint regression branches to the original YOLOv5 structure, enabling rotating target detection and corner regression. Furthermore, Wing loss can be used instead of L1 loss to improve the accuracy of keypoint regression.

[0094] The table image recognition device provided by the present invention is described below. The table image recognition device described below can be referred to in correspondence with the table image recognition method described above.

[0095] This invention proposes a table image recognition device, comprising:

[0096] The semantic segmentation module is used to mark the table line pixels in the table image to be recognized, and obtain a semantic segmentation binarized image.

[0097] The table line position determination module is used to sequentially perform image correction, table line extraction, and table line position determination on the semantic segmentation binarized image, and determine the position coordinates of the table lines.

[0098] The cell determination module is used to determine the start and end rows and columns of a cell based on the position coordinates of the table lines.

[0099] The character segmentation module is used to segment the table image to be recognized into single characters and determine the character position information, wherein the character position information represents the position information of each character.

[0100] The aggregation module is used to aggregate the cell and its corresponding text content based on the start and end rows and the character position information, and generate a table file.

[0101] Furthermore, the table line position determination module is also used for:

[0102] Based on the semantic segmentation binarized graph, the connected regions in the semantic segmentation binarized graph are extracted;

[0103] Fit the bounding lines of the connected regions to determine the table lines to be identified;

[0104] The position coordinates of the table line are determined based on the smallest bounding rectangle corresponding to the table line to be identified.

[0105] Furthermore, the character segmentation module is also used for:

[0106] The image of the table to be recognized is input into a convolutional recurrent neural network to obtain a probability matrix of the sequence vector length and the number of characters output by the convolutional recurrent neural network.

[0107] The probability matrix is ​​used to determine the character segmentation position and obtain the character position information. If the character is a Chinese character, the index of the first empty sequence vector is determined as the starting index of the Chinese character, and the starting index of the Chinese character is used to determine the starting coordinates of the Chinese character. If the character is a character other than a Chinese character, the index of the second empty sequence vector is determined as the starting index of the other character, and the starting index of the other character is used to determine the starting coordinates of the other character. The starting coordinates are used to determine the character segmentation position.

[0108] Furthermore, the aggregation module is also used for:

[0109] Input the start and end rows and columns and the character position information into the aggregation model, aggregate the character boxes and cell boxes, and generate a table file;

[0110] The aggregation model is used to match text content with corresponding cells, and the algorithms applied by the aggregation model include the central dogma and the maximum IOU rule.

[0111] Furthermore, the table line position determination module is also used for:

[0112] The semantic segmentation binarized image is input into the rotating target detection model of the corner regression algorithm to determine the table region and table corner of the semantic segmentation binarized image;

[0113] The table area and the table corner points are used to perform image correction on the semantic segmentation binarized image.

[0114] Furthermore, the rotating target detection model is an improved network based on the YOLOv5 network. The improved network structure is an angle category branch and a key point regression branch added to the structure of the YOLOv5 network.

[0115] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a table image recognition method. This method includes: marking table lines in the table image to be recognized with pixels to obtain a semantic segmentation binarized image; sequentially performing image correction, table line extraction, and table line position determination on the semantic segmentation binarized image to determine the position coordinates of the table lines; determining the start and end rows and columns of a cell based on the position coordinates of the table lines; performing single-character position segmentation on the table image to be recognized to determine character position information, wherein the character position information represents the position information of each character; and aggregating the cell with the corresponding text content based on the start and end rows and the character position information to generate a table file.

[0116] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the table image recognition method provided by the above methods. The method includes: marking table lines in a table image to be recognized with pixels to obtain a semantic segmentation binarized image; performing image correction, table line extraction, and table line position determination on the semantic segmentation binarized image in sequence to determine the position coordinates of the table lines; determining the start and end rows and columns of a cell based on the position coordinates of the table lines; performing single-character position segmentation on the table image to be recognized to determine character position information, wherein the character position information represents the position information of each character; and aggregating the cell with the corresponding text content based on the start and end rows and the character position information to generate a table file.

[0118] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the table image recognition method provided by the above methods. The method includes: marking table lines in a table image to be recognized with pixels to obtain a semantic segmentation binarized image; sequentially performing image correction, table line extraction, and table line position determination on the semantic segmentation binarized image to determine the position coordinates of the table lines; determining the start and end rows and columns of a cell based on the position coordinates of the table lines; performing single-character position segmentation on the table image to be recognized to determine character position information, wherein the character position information represents the position information of each character; and aggregating the cell with the corresponding text content based on the start and end rows and the character position information to generate a table file.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing table images, characterized in that, include: The table image to be recognized is marked with table line pixels to obtain a semantic segmentation binarized image; The semantic segmentation binarized image is sequentially subjected to image correction, table line extraction, and table line position determination to determine the position coordinates of the table lines; Based on the position coordinates of the table lines, determine the start and end rows and columns of the cells; The table image to be identified is segmented into single characters to determine the character position information, which represents the position information of each character. Based on the start and end rows and columns and the character position information, the cells and their corresponding text content are aggregated to generate a table file; The table image to be identified is segmented into single characters to determine character position information, including: The image of the table to be recognized is input into a convolutional recurrent neural network to obtain a probability matrix of the sequence vector length and the number of characters output by the convolutional recurrent neural network. The probability matrix is ​​used to determine the character segmentation position and obtain the character position information. If the character is a Chinese character, the index of the first empty sequence vector is determined as the starting index of the Chinese character, and the starting index of the Chinese character is used to determine the starting coordinates of the Chinese character. If the character is a character other than a Chinese character, the index of the second empty sequence vector is determined as the starting index of the other character, and the starting index of the other character is used to determine the starting coordinates of the other character. The starting coordinates are used to determine the character segmentation position. The first empty sequence vector is the first few empty sequences of the sequence vector corresponding to Chinese characters, used to adjust the positioning of Chinese characters; the second empty sequence vector is the first few empty sequences of the sequence vector corresponding to other characters, used to adjust the positioning of other characters; the determination of the first and second empty sequence vectors includes determining the length and type of the recognition sequence vector; if the length of the sequence vector exceeds a set threshold, the empty sequence vector is incremented by 0.5, otherwise it remains unchanged; if the type of the sequence vector is a Chinese sequence, the empty sequence vector is incremented by 1.

2. The table image recognition method according to claim 1, characterized in that, The semantic segmentation binarized image is subjected to table line extraction and table line position determination. The position coordinates of the table lines are determined, including: Based on the semantic segmentation binarized graph, the connected regions in the semantic segmentation binarized graph are extracted; Fit the bounding lines of the connected regions to determine the table lines to be identified; The position coordinates of the table line are determined based on the smallest bounding rectangle corresponding to the table line to be identified.

3. The table image recognition method according to claim 1, characterized in that, Based on the start and end rows and columns and the character position information, the cells and their corresponding text content are aggregated to generate a table file, including: Input the start and end rows and columns and the character position information into the aggregation model, aggregate the character boxes and cell boxes, and generate a table file; The aggregation model is used to match text content with corresponding cells, and the algorithms applied by the aggregation model include the central dogma and the maximum IOU rule.

4. The table image recognition method according to claim 1, characterized in that, Image correction is performed on the semantic segmentation binarized image, including: The semantic segmentation binarized image is input into the rotating target detection model of the corner regression algorithm to determine the table region and table corner of the semantic segmentation binarized image; The table area and the table corner points are used to perform image correction on the semantic segmentation binarized image.

5. The table image recognition method according to claim 4, characterized in that, The rotating target detection model is an improved version of the YOLOv5 network. The improved network structure is an angle category branch and a key point regression branch added to the YOLOv5 network structure.

6. A table image recognition device, characterized in that, The table image recognition method as described in claim 1 includes: The semantic segmentation module is used to mark the table line pixels in the table image to be recognized, and obtain a semantic segmentation binarized image. The table line position determination module is used to sequentially perform image correction, table line extraction, and table line position determination on the semantic segmentation binarized image, and determine the position coordinates of the table lines. The cell determination module is used to determine the start and end rows and columns of a cell based on the position coordinates of the table lines. The character segmentation module is used to segment the table image to be recognized into single characters and determine the character position information, wherein the character position information represents the position information of each character. The aggregation module is used to aggregate the cell and its corresponding text content based on the start and end rows and the character position information to generate a table file.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the table image recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the table image recognition method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the table image recognition method as described in any one of claims 1 to 5.

Images