A table structure identification method and device, a storage medium and an electronic device
By using a key point prediction and matching model and heatmap technology to identify table structure, the problem of insufficient accuracy in table structure identification in existing technologies is solved, and accurate reconstruction of tables with missing borders is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN QIANHAI HUANRONG LIANYI INFORMATION TECHNOLOGY SERVICES CO LTD
- Filing Date
- 2022-11-30
- Publication Date
- 2026-07-24
AI Technical Summary
In existing technologies, table structure recognition methods have high requirements for the integrity of the frame lines, cannot completely reconstruct tables with missing frame lines, and have errors in cell coordinate positioning, resulting in insufficient recognition accuracy.
A key point prediction model is used to predict the coordinates and type information of each key point through a heat map. A key point matching model is then used to match the key points and construct a set of key points. The table structure is then reconstructed based on these key point sets.
It achieves accurate recognition of table structure, avoids strict dependence on table lines, and can accurately reconstruct tables with missing borders, thus improving recognition accuracy.
Smart Images
Figure CN115965985B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for recognizing table structures. Background Technology
[0002] Tables, as a structured data representation, appear extensively in everyday documents. Table structure recognition can digitize paper documents, converting them into corresponding electronic spreadsheets for easier statistics and storage.
[0003] Existing methods for table structure recognition typically employ image segmentation schemes based on traditional digital image processing techniques or deep learning to extract the horizontal and vertical lines of the table, and then obtain the coordinates of each cell within the table through post-processing methods. Because this method is based on table line extraction, it requires extremely high integrity of the table borders and cannot completely reconstruct tables with missing borders. Furthermore, the method of segmenting first and then extracting coordinates after processing introduces errors in cell coordinate localization, resulting in inaccurate table structure recognition.
[0004] Therefore, there is an urgent need for a table structure recognition method to solve the problem that the existing technology is not accurate enough in table structure recognition. Summary of the Invention
[0005] In view of this, the present invention provides a table structure recognition method, apparatus, storage medium and device, the main purpose of which is to solve the problem of insufficient accuracy in the current table structure recognition.
[0006] To address the aforementioned problems, this application provides a method for recognizing table structures, comprising:
[0007] Retrieve the target image containing the target table object;
[0008] Using a predetermined target key point prediction model and a key point heatmap prediction method, the target image is used to predict key points and obtain the coordinate values and type information of each key point.
[0009] Using a predetermined target key point matching model, based on the coordinate values and type information of each key point, the key points are matched to obtain several key point sets.
[0010] The table is reconstructed based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0011] Optionally, obtaining the target image containing the target table object includes:
[0012] Receive the original image input by the user;
[0013] Based on a predetermined feature extraction model, table features are extracted from the original image to obtain the target image containing the target table object.
[0014] Optionally, the keypoint prediction model uses a keypoint heatmap prediction method to predict keypoints in the target image, obtaining the coordinate values of each keypoint and the type information of each keypoint, including:
[0015] Based on the key point prediction model, the target image is predicted using a key point heatmap prediction method to obtain a heatmap image containing several key points corresponding to various types of information; the type information includes: top left corner point, top right corner point, bottom right corner point, bottom left corner point, and center point;
[0016] Based on the thermal images, the coordinate values and type information of each key point in each thermal image are obtained.
[0017] Optionally, the keypoint matching model is used to match each keypoint based on its coordinate values and type information to obtain a set of keypoints, including:
[0018] Based on the coordinate values of each key point, the affinity between each key point is calculated using the key point matching model.
[0019] Based on the affinity between key points and the type information of each key point, at least three key points with affinity higher than a predetermined value and different type information are obtained, and a key point set is constructed to obtain the aforementioned key point set.
[0020] Optionally, the step of calculating the affinity between key points based on their coordinate values using the key point matching model includes:
[0021] Based on the coordinate values of each key point, the key point matching model is used to map each key point to the target feature space to obtain the mapped key points of each key point in the target feature space.
[0022] Obtain the feature vectors of each of the mapping key points;
[0023] The difference between each key point is calculated based on the feature vectors to obtain the affinity between each key point.
[0024] Optionally, before obtaining the target image containing the target table object, the method further includes: training a target keypoint prediction model, including:
[0025] Obtain several sample table images and the coordinate labels of each key point in each sample table image;
[0026] Based on the aforementioned sample table images, the initial keypoint prediction model is used to predict keypoints, thereby obtaining the predicted coordinate values of several predicted keypoints corresponding to each of the sample table images.
[0027] Based on the predicted coordinate values and coordinate labels, the model parameters in the initial keypoint prediction model are adjusted to obtain the target keypoint prediction model.
[0028] Optionally, before obtaining the target image containing the target table object, the method further includes: training the keypoint matching model, including:
[0029] Obtain several sets of key points for samples and the coordinate values of each key point in each set of key points;
[0030] Based on the coordinate values of each sample key point, the affinity between each sample key point is calculated using the initial key point matching model.
[0031] Based on the affinity between key points of each sample and the type information of key points of each sample, at least three key points with affinity higher than a predetermined value and different type information are obtained to construct a first key point set, thereby obtaining the plurality of first key point sets;
[0032] Based on each of the first keypoint sets and each of the sample keypoint sets, the model parameters in the initial keypoint matching model are adjusted to obtain the target keypoint matching model.
[0033] To address the aforementioned problems, this application provides a table structure recognition device, comprising:
[0034] The acquisition module is used to acquire the target image containing the target table object;
[0035] The prediction module is used to predict key points in the target image by using a predetermined target key point prediction model and a key point heatmap prediction method, so as to obtain the coordinate values of each key point and the type information of each key point.
[0036] The matching module is used to match each key point based on the coordinate value and type information of each key point using a predetermined target key point matching model to obtain a set of key points.
[0037] The reconstruction module is used to reconstruct the table based on the coordinate values in each of the key point sets, so as to obtain the recognition result of the target table object.
[0038] To address the aforementioned problems, this application provides a storage medium storing a computer program that, when executed by a processor, implements the steps of the table structure recognition method described above.
[0039] To address the aforementioned problems, this application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of the table structure recognition method described above.
[0040] This application discloses a table structure recognition method, apparatus, storage medium, and electronic device. By utilizing a key point prediction model and employing a heatmap prediction method, it can accurately predict each key point. By using a key point matching model, it can accurately match each key point to obtain several key point sets corresponding to each cell. Subsequently, it can accurately reconstruct the target table object based on each key point set.
[0041] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0043] Figure 1 This is a flowchart illustrating a table structure recognition method according to an embodiment of this application;
[0044] Figure 2 This is a flowchart illustrating a table structure recognition method according to another embodiment of this application;
[0045] Figure 3 This is a structural block diagram of a table structure recognition device according to another embodiment of this application;
[0046] Figure 4 This is a structural block diagram of an electronic device according to another embodiment of this application. Detailed Implementation
[0047] Various embodiments and features of this application are described herein with reference to the accompanying drawings.
[0048] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.
[0049] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0050] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0051] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.
[0052] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0053] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.
[0054] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.
[0055] This application provides a method for recognizing table structures, which can be specifically applied to electronic devices such as terminals and servers. Figure 1 As shown, the method in this embodiment includes:
[0056] Step S101: Obtain the target image containing the target table object;
[0057] In this step, a feature extraction model is used to extract table features from the original image to obtain a target image containing a specific table object. In other words, the original image may contain multiple table objects, or it may contain both tables and text paragraphs. In such cases, the table needs to be extracted from the original image to remove redundant text paragraphs, laying the foundation for subsequent table recognition and reconstruction of the target image containing only table objects. The feature extraction model used in this step can be a deep residual convolutional neural network (ResNet).
[0058] Step S102: Using a predetermined target key point prediction model, the key points of the target image are predicted by using a key point heatmap prediction method to obtain the coordinate values of each key point and the type information of each key point.
[0059] In the specific implementation of this step, a target keypoint prediction model can be pre-trained. This model is then used to accurately predict keypoints. Specifically, it involves acquiring heatmap information of the keypoints to obtain heatmaps containing several top-left corner points, top-right corner points, bottom-right corner points, bottom-left corner points, and a center point. These five different types of heatmaps are then used to obtain the coordinate values of each keypoint within them. The type information refers to the type of keypoint, including top-left, top-right, bottom-right, bottom-left, and center points.
[0060] Step S103: Using a predetermined target key point matching model, based on the coordinate values of each key point and the type information of each key point, match each key point to obtain a set of several key points.
[0061] In this step, after obtaining the coordinates of each keypoint, the affinity between keypoints can be calculated based on their coordinates. Then, the affinity is used to determine whether two keypoints are located in the same cell. When the affinity is high (e.g., greater than a predetermined value) and the two keypoints are of different types, it indicates that two keypoints, such as keypoint A (top left corner type) and keypoint B (center point type), are located in the same cell. Further investigation can be conducted to determine if there are other types of keypoints with high affinity to keypoint A or keypoint B. For example, keypoint C (top right corner type) with affinity greater than a predetermined value to keypoint B can be identified. Based on keypoints A, B, and C, a keypoint set can then be constructed.
[0062] Step S104: Reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0063] In the specific implementation process of this step, after obtaining several sets of key points, the cells corresponding to the sets can be drawn based on the position coordinates of the key points in the sets. Then, the cells corresponding to each set are combined to reconstruct the target table object.
[0064] The table structure recognition method in this embodiment utilizes a key point prediction model and a heatmap prediction method to accurately predict each key point. By employing a key point matching model, it can precisely match each key point, obtaining several key point sets corresponding to each cell. This allows for accurate reconstruction of the target table object based on these key point sets. It eliminates the need for line condition recognition within the table, solving the problems of inaccurate recognition results due to blurred lines.
[0065] Another embodiment of this application provides a method for recognizing table structures, including:
[0066] Step S201: Train and obtain the target key point prediction model.
[0067] In this step, the training process of the target keypoint prediction model is as follows: acquire several sample table images and coordinate labels of each keypoint in each sample table image; perform keypoint prediction using the initial keypoint prediction model based on the several sample table images to obtain the predicted coordinate values of several predicted keypoints corresponding to each sample table image; adjust the model parameters in the initial keypoint prediction model based on each predicted coordinate value and each coordinate label to obtain the target keypoint prediction model.
[0068] Step S202: Train to obtain the key point matching model;
[0069] In this step, the training process of the target keypoint matching model is as follows: Several sample keypoint sets and the coordinate values of each sample keypoint in each sample keypoint set are obtained; based on the coordinate values of each sample keypoint, the affinity between each sample keypoint is calculated using an initial keypoint matching model; based on the affinity between each sample keypoint and the type information of each sample keypoint, at least three keypoints with affinity higher than a predetermined value and different type information are obtained to construct a first keypoint set, thereby obtaining the several first keypoint sets; based on each first keypoint set and each sample keypoint set, the model parameters in the initial keypoint matching model are adjusted to obtain the target keypoint matching model.
[0070] Step S203: Receive the original image input by the user; based on a predetermined feature extraction model, perform table feature extraction on the original image to obtain the target image containing the target table object;
[0071] In this step, the feature extraction model can specifically be a deep residual convolutional neural network, ResNet.
[0072] In the specific implementation process, a feature extraction model can be pre-trained, which involves using several sample images and label images containing sample table objects corresponding to the original images of each sample. Then, the initial feature extraction model is used to extract features from the sample images to obtain the extracted images. Each extracted image is then compared with the label images. If the difference between the two is large, the model parameters in the initial feature extraction model are adjusted to obtain the target feature extraction model.
[0073] By using a feature extraction model to extract table features from the original image, redundant text paragraphs or other content unrelated to the table object can be removed, thus providing a guarantee for subsequent table recognition and reconstruction of target images containing only table objects.
[0074] Step S204: Based on the key point prediction model, the target image is predicted using a key point heatmap prediction method to obtain a heatmap image containing several key points corresponding to various types of information; the type information includes: top left corner point, top right corner point, bottom right corner point, bottom left corner point, and center point;
[0075] In the specific implementation process of this step, for different key point types, corresponding thermal images can be predicted and obtained respectively. That is, the first thermal image corresponding to the upper left corner point, the second thermal image corresponding to the upper right corner point, the third thermal image corresponding to the lower right corner point, the fourth thermal image corresponding to the lower left corner point, and the fifth thermal image corresponding to the center point can be predicted and obtained.
[0076] Step S205: Based on each of the thermal images, obtain the coordinate values and type information of each key point in each of the thermal images;
[0077] In this step, after obtaining each thermal image, a common coordinate system can be established based on each thermal image, and then the coordinate values of each key point in each thermal image can be obtained based on this coordinate system.
[0078] Step S206: Based on the coordinate values of each key point, calculate the affinity between each key point using the key point matching model;
[0079] In the specific implementation process of this step, based on the coordinate values of each key point, the key point matching model can be used to map each key point to the target feature space to obtain the mapped key points of each key point in the target feature space; the feature vector of each mapped key point can be obtained; and the difference between each key point can be calculated based on each feature vector to obtain the affinity between each key point.
[0080] In the specific implementation process, to make the affinity calculation more reasonable and accurate, so as to facilitate the subsequent matching of key points corresponding to the same cell and constructing the corresponding key point set, this step can use the center point as the benchmark to calculate the affinity between each other key point and each center point separately. That is, calculate the affinity between each top-left corner point and each center point, each top-right corner point and each center point, each bottom-right corner point and each center point, and each bottom-left corner point and each center point. The specific principle of affinity calculation also adopts the above method, that is, mapping each key point to the target feature space to obtain the mapped key points of each key point in the target feature space; and obtaining the feature vector of each mapped key point. The difference between keypoints of non-center point type and keypoints of center point type is calculated based on the feature vectors of each keypoint to obtain the affinity between keypoints; that is, based on the feature vectors corresponding to each center point, the feature vectors corresponding to the top left corner point, the top right corner point, the bottom right corner point, and the bottom left corner point, the affinity between each top left corner point and each center point, the affinity between each top right corner point and each center point, the affinity between each bottom right corner point and each center point, and the affinity between each bottom left corner point and each center point are calculated respectively.
[0081] Step S207: Based on the affinity between each key point and the type information of each key point, at least three key points with affinity higher than a predetermined value and different type information are obtained, and a key point set is constructed to obtain the set of several key points;
[0082] In this step, the predetermined value of affinity can be set according to actual needs, such as 75, 80, or 85. After calculating the affinity between each keypoint, this step can determine whether two keypoints belong to the same cell based on the affinity level. To make the final matching result more accurate, that is, to accurately match the vertices and center points of the same cell into the same set, this step can also combine the type information of each keypoint. That is, after determining that the affinity is greater than the predetermined value, it further determines whether the key types are the same, avoiding matching two keypoints of the upper right corner type into the same set, thus preventing inaccurate reconstruction of subsequent cells.
[0083] In practical implementation, when the calculated affinity is the affinity between non-center point type keypoints and center point type keypoints, a keypoint set corresponding to each center point can be constructed. In this embodiment, since each cell corresponds to one center point, by constructing a keypoint set corresponding to the center point, the vertices corresponding to each cell can be assigned to the same keypoint set, laying the foundation for subsequent accurate cell reconstruction based on each keypoint set.
[0084] Step S208: Reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0085] In the specific implementation process of this step, after obtaining several sets of key points, the position coordinates of the key points in the key point sets can be used to draw the cells corresponding to the key point sets. Then, the cells corresponding to each key point set are spliced and combined to reconstruct the target table object and obtain the recognition result of the table structure.
[0086] The table structure recognition method in this embodiment accurately predicts key points using a key point prediction model and a heatmap prediction method. By employing a key point matching model, it precisely matches each key point, obtaining several key point sets corresponding to each cell. This allows for accurate reconstruction of the target table object based on these key point sets. This application avoids the heavy reliance on complete table lines in existing table line extraction schemes by extracting key point information from the table, achieving good results even for cells with missing borders. Furthermore, the table recognition scheme based on key point detection in this invention only requires visual information from the image, resulting in relatively low data dependence.
[0087] Based on the above embodiments, the solution in this application will be described below in conjunction with specific application scenarios. For example... Figure 2 As shown, the method in this embodiment includes the following steps:
[0088] Step 1: Input the table image into a deep residual convolutional neural network (ResNet) for feature extraction to obtain the feature information of the input image;
[0089] Step 2: Using the key point prediction branch model, predict the cell key points and the relationships between key points based on the features extracted by ResNet, and obtain the key point coordinate information.
[0090] In this embodiment, since directly regressing cell vertex coordinates using a deep learning model is generally difficult and has low accuracy, and since the number of cells in different tables is not fixed, it is impossible to directly predict cell coordinates using a fully connected network. Therefore, this embodiment indirectly obtains point coordinates by predicting a keypoint heatmap. The keypoint prediction principle is as follows: the keypoint information of the cells is plotted as a heatmap on a blank H×W map. The keypoint prediction branch model only needs to predict the H×W heatmap. Furthermore, to facilitate subsequent prediction of matching relationships between keypoints, the keypoint prediction branch model, in addition to predicting the four vertices of the cell, also predicts the cell's center point. Therefore, the output feature map H of this branch is H×W×5, where the five channels of the feature map predict the center points of all cells in the table sequentially. Top left corner Top right corner bottom right corner and the bottom left corner
[0091] Step 3: Using the key point matching branch model, the key points are grouped based on their coordinate information and the relationships between them, resulting in several key point sets, which in turn yields the coordinate information of each cell.
[0092] In this embodiment, directly finding the relationship between the four vertices of a cell is difficult. Therefore, this embodiment uses the cell center point as a bridge to construct two-point relationships (i.e., the relationship between the center point and the top-left corner, the center point and the top-right corner, the center point and the bottom-right corner, and the center point and the bottom-left corner), thereby matching and obtaining the vertices of the same cell. Specifically, the principle of keypoint matching is: mapping all keypoints to the same feature space, making the distance between two points belonging to the same cell as close as possible in space, and making the distance between two points not belonging to the same cell as far as possible in space. A feature map M of size H×W×5 is predicted using the keypoint matching branch model. This feature map can be considered as the mapping of the center point, top-left corner, top-right corner, bottom-right corner, and bottom-left corner onto another feature space V. This dimensional space can represent the affinity relationship of keypoints at corresponding positions. By locating the coordinates of the cell's center point, top-left corner, top-right corner, bottom-right corner, and bottom-left corner through the keypoint prediction branch, and then using the obtained keypoint coordinates... Obtain the feature vector of the point in the feature map M. Then calculate the feature vector of the center point respectively. With vertex eigenvectors The difference between the center point and the top left, top right, bottom right, and bottom left corner points in the feature space V is found. The smaller the difference, the higher the affinity. That is, the center point and the point together form a cell. Therefore, the center point can establish a relationship with the other four vertices through keypoint matching. The four vertices that establish a relationship with the same center point form a cell, thereby constructing a set of keypoints.
[0093] Step 4: Reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0094] In this step, after locating the center point of all cells and obtaining the set of key points corresponding to each center, the coordinates of all cells in the table can be further determined, each cell can be reconstructed, and the target table can be reconstructed.
[0095] This embodiment utilizes a keypoint prediction model and a heatmap-based prediction method to accurately predict each keypoint. By employing a keypoint matching model, it can precisely match each keypoint, obtaining several keypoint sets corresponding to each cell. Subsequently, the target table object can be accurately reconstructed based on these keypoint sets. This eliminates the need for line condition recognition within the table, solving the problems of inaccurate recognition results due to blurred lines.
[0096] Another embodiment of this application provides a table structure recognition device, such as... Figure 3 As shown, it includes:
[0097] Module 11 is used to acquire the target image containing the target table object;
[0098] Prediction module 12 is used to predict key points of the target image by using a predetermined target key point prediction model and key point heatmap prediction method, and to obtain the coordinate values of each key point and the type information of each key point.
[0099] Matching module 13 is used to match each key point based on the coordinate value of each key point and the type information of each key point using a predetermined target key point matching model to obtain a set of key points.
[0100] The reconstruction module 14 is used to reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0101] In this embodiment, the acquisition module specifically includes a receiving unit and a feature extraction unit. The receiving unit is used to: receive the original image input by the user;
[0102] The feature extraction unit is used to: extract table features from the original image based on a predetermined feature extraction model, so as to obtain the target image containing the target table object.
[0103] In this embodiment, the prediction module includes a prediction unit and an acquisition unit. The prediction unit is used to: predict the key points of the target image using a key point heatmap prediction method based on a key point prediction model, and obtain a heatmap image containing several key points corresponding to various types of information; the type information includes: top left corner point, top right corner point, bottom right corner point, bottom left corner point, and center point;
[0104] The obtaining unit is used to: obtain the coordinate values and type information of each key point in each of the thermal images based on each of the thermal images.
[0105] In this embodiment, the matching module includes a calculation unit and a construction unit. The calculation unit is used to calculate the affinity between key points based on the coordinate values of each key point using the key point matching model.
[0106] The construction unit is used to: based on the affinity between each key point and the type information of each key point, obtain at least 3 key points with affinity higher than a predetermined value and different type information, construct a key point set, and obtain the several key point sets.
[0107] In this embodiment, the calculation unit is configured to: map each key point to a target feature space based on the coordinate values of each key point using the key point matching model, thereby obtaining mapped key points of each key point in the target feature space; obtain feature vectors of each mapped key point; and calculate the difference between each key point based on the feature vectors to obtain the affinity between each key point.
[0108] In this embodiment, the table structure recognition device further includes a first training model for training and obtaining the target key point prediction model. The first training module is specifically used for: acquiring several sample table images and coordinate labels of key points in each sample table image; predicting key points based on the several sample table images using an initial key point prediction model to obtain predicted coordinate values of several predicted key points corresponding to each sample table image; and adjusting the model parameters in the initial key point prediction model based on each predicted coordinate value and each coordinate label to obtain the target key point prediction model.
[0109] In this embodiment, the table structure recognition device further includes a second training model for training the keypoint matching model. The second training module is specifically used for: acquiring several sample keypoint sets and the coordinate values of each sample keypoint in each sample keypoint set; calculating the affinity between each sample keypoint using an initial keypoint matching model based on the coordinate values of each sample keypoint; acquiring at least three keypoints with affinity higher than a predetermined value and different type information based on the affinity between each sample keypoint and the type information of each sample keypoint, constructing a first keypoint set to obtain the several first keypoint sets; and adjusting the model parameters in the initial keypoint matching model based on each first keypoint set and each sample keypoint set to obtain the target keypoint matching model.
[0110] The table structure recognition device in this embodiment accurately predicts key points using a key point prediction model and a heatmap prediction method. By employing a key point matching model, it precisely matches each key point, obtaining several sets of key points corresponding to each cell. This allows for accurate reconstruction of the target table object based on these key point sets. It eliminates the need for line condition recognition within the table, solving the problems of inaccurate recognition results due to blurred lines.
[0111] Another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the following method steps:
[0112] Step 1: Obtain the target image containing the target table object;
[0113] Step 2: Using a predetermined target key point prediction model, the key point heatmap prediction method is used to predict the key points of the target image, and obtain the coordinate values of each key point and the type information of each key point.
[0114] Step 3: Using a predetermined target key point matching model, match each key point based on its coordinate values and type information to obtain several key point sets.
[0115] Step 4: Reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0116] The specific implementation process of the above method steps can be found in the embodiment of the above arbitrary table structure recognition method, which will not be repeated here.
[0117] The storage medium in this application can accurately predict each key point by using a key point prediction model and a heatmap prediction method. By using a key point matching model, each key point can be accurately matched to obtain several key point sets corresponding to each cell. Subsequently, the target table object can be accurately reconstructed based on each key point set.
[0118] Another embodiment of this application provides an electronic device, such as... Figure 4 As shown, it includes at least a memory 1 and a processor 2. The memory 1 stores a computer program, and the processor 2 performs the following method steps when executing the computer program in the memory 1:
[0119] Step 1: Obtain the target image containing the target table object;
[0120] Step 2: Using a predetermined target key point prediction model, the key point heatmap prediction method is used to predict the key points of the target image, and obtain the coordinate values of each key point and the type information of each key point.
[0121] Step 3: Using a predetermined target key point matching model, match each key point based on its coordinate values and type information to obtain several key point sets.
[0122] Step 4: Reconstruct the table based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
[0123] The specific implementation process of the above method steps can be found in the embodiment of the above arbitrary table structure recognition method, which will not be repeated here.
[0124] The storage medium in this application can accurately predict each key point by using a key point prediction model and a heatmap prediction method. By using a key point matching model, each key point can be accurately matched to obtain several key point sets corresponding to each cell. Subsequently, the target table object can be accurately reconstructed based on each key point set.
[0125] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
Claims
1. A method for recognizing table structures, characterized in that, include: The target image containing the target table object is obtained, and the feature information of the target image is obtained by performing feature extraction on the target image based on the deep residual convolutional neural network ResNet. Based on the feature information of the target image, using a predetermined target key point prediction model, the key point information of the cell is plotted in the form of a heatmap on a blank H×W map to obtain a feature map containing 5 channels. Based on the feature map, the coordinate values of each key point and the type information of the key point are obtained. The 5 channels of the feature map correspond one-to-one with the type information. The 5 channels of the feature map predict the center point, top left corner point, top right corner point, bottom right corner point and bottom left corner point of all cells in the table in sequence. Based on the coordinate values of each key point, each key point is mapped to the target feature space using a predetermined target key point matching model, thereby obtaining the mapped key points of each key point in the target feature space. Obtain the feature vectors of each of the mapping key points; Based on the feature vectors described above, the difference between key points of each non-center point type and key points of the center point type is calculated to obtain the affinity between key points; Based on the affinity between key points and the type information of each key point, at least three key points with affinity higher than a predetermined value and different type information are obtained, and a set of key points corresponding to each center point is constructed to obtain several sets of key points. The table is reconstructed based on the coordinate values in each set of key points to obtain the recognition result of the target table object.
2. The method according to claim 1, characterized in that, The step of obtaining the target image containing the target table object includes: Receive the original image input by the user; Based on a predetermined feature extraction model, table features are extracted from the original image to obtain the target image containing the target table object.
3. The method according to claim 1, characterized in that, Obtain the coordinate values of each key point and the type information of each key point, including: Based on the key point prediction model, the key point heatmap prediction method is used to predict the key points of the target image to obtain a heatmap image containing several key points corresponding to each type of information. Based on the thermal images, the coordinate values and type information of each key point in each thermal image are obtained.
4. The method according to claim 1, characterized in that, Before acquiring the target image containing the target table object, the method further includes: training a target keypoint prediction model, including: Obtain several sample table images and the coordinate labels of each key point in each sample table image; Based on the aforementioned sample table images, the initial keypoint prediction model is used to predict keypoints, thereby obtaining the predicted coordinate values of several predicted keypoints corresponding to each of the sample table images. Based on the predicted coordinate values and coordinate labels, the model parameters in the initial keypoint prediction model are adjusted to obtain the target keypoint prediction model.
5. The method according to claim 1, characterized in that, Before obtaining the target image containing the target table object, the method further includes: training the keypoint matching model, including: Obtain several sets of sample key points and the coordinate values of each sample key point in each set of sample key points; Based on the coordinate values of each sample key point, the affinity between each sample key point is calculated using the initial key point matching model. Based on the affinity between key points of each sample and the type information of key points of each sample, at least three key points with affinity higher than a predetermined value and different type information are obtained to construct a first key point set, thereby obtaining several first key point sets. Based on each of the first keypoint sets and each of the sample keypoint sets, the model parameters in the initial keypoint matching model are adjusted to obtain the target keypoint matching model.
6. A table-structure recognition device, characterized in that, include: The acquisition module is used to acquire the target image containing the target table object, and to extract features from the target image based on the deep residual convolutional neural network ResNet to obtain the feature information of the target image; The prediction module is used to plot the key point information of cells in the form of a heatmap on a blank H×W map based on the feature information of the target image and a predefined target key point prediction model. This results in a feature map containing 5 channels. The coordinate values and type information of each key point are obtained based on the feature map. The 5 channels of the feature map correspond one-to-one with the type information. The 5 channels of the feature map predict the center point, top left corner point, top right corner point, bottom right corner point, and bottom left corner point of all cells in the table in sequence. The matching module is used to map each key point to a target feature space based on the coordinate values of each key point using a predetermined target key point matching model, thereby obtaining mapped key points of each key point in the target feature space; obtain feature vectors of each mapped key point; calculate the difference between each non-center point type key point and center point type key point based on the feature vectors, thereby obtaining the affinity between each key point; and, based on the affinity between each key point and the type information of each key point, obtain at least three key points with affinity higher than a predetermined value and different type information, and construct a key point set corresponding to each center point, thereby obtaining several key point sets. The reconstruction module is used to reconstruct the table based on the coordinate values in each of the key point sets, so as to obtain the recognition result of the target table object.
7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the table structure recognition method according to any one of claims 1-5.
8. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of the table structure recognition method according to any one of claims 1-5.