Table structuring method, related equipment and program product

Abstract

The invention discloses a table structuring method, related equipment and a program product, and relates to the technical field of computer vision and table recognition. The method comprises the following steps: inputting a to-be-structured table image into a configured control parameter prediction model to obtain a group of control parameters respectively corresponding to each row and column line in the table image; generating a curvilinear function corresponding to each row and column line based on the group of control parameters corresponding to each row and column line and a preset curvilinear function containing a group of unknown control parameters; and generating a structured table of the table image according to the curvilinear function corresponding to each row and column line. The model directly predicts the function representation of each row and column line, and eliminates the additional error caused by the fitting process, thereby improving the accuracy of the structured table finally generated by the curvilinear function of the row and column lines.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and table recognition, and more particularly, to a table structuring method, related equipment and program product. BACKGROUND

[0002] As a form of efficient combination and presentation of data, tables are widely used in various documents, such as office documents, reports, web pages, etc. However, these table data often exist in unstructured or semi-structured forms, which is not conducive to efficient retrieval, analysis and management by computers. Therefore, there is an urgent need for a table structuring method to convert these complex forms of table data into unified and standardized structured data, so that computers can conveniently process them.

[0003] CN119992578A discloses a table recognition method, which extracts row line key points and column line key points in a table image based on table row features and table column features of the table image, respectively, and fits row division lines and column division lines according to the row line key points and the column line key points, respectively, to generate a structured table. In the key point fitting process, the fitting algorithm introduces fitting errors, resulting in low accuracy of the structured table finally recognized.

[0004] Therefore, there is an urgent need for a method for accurately recognizing tables in images to structure tables. SUMMARY

[0005] In view of the above problems, the present application is proposed to provide a table structuring method, related equipment and program product to improve the accuracy of table recognition. The specific scheme is as follows:

[0006] In a first aspect, a table structuring method is provided, comprising:

[0007] obtaining a table image to be structured and a preset curve function comprising a group of unknown control parameters, the control parameters being used to control the shape and position of the curve represented by the curve function;

[0008] inputting the table image into a configured control parameter prediction model to obtain a group of control parameters corresponding to each row and column line in the table image, the control parameter prediction model being trained by table image sample data labeled with a group of control parameters corresponding to each row and column line;

[0009] generating a curve function corresponding to each row and column line based on the group of control parameters corresponding to each row and column line and the curve function;

[0010] generating a structured table of the table image according to the curve function corresponding to each row and column line.

[0011] In one possible design, in another implementation of the first aspect of the embodiments of this application, the training process of the control parameter prediction model includes:

[0012] Obtain a table image sample and its corresponding control parameter labels, wherein the control parameter labels include a set of control parameters corresponding to each row and column line in the table image sample;

[0013] Extract image features from the table image samples, and predict control parameters based on the image features;

[0014] Calculate the first loss value based on the prediction results of the control parameters and the control parameter labels;

[0015] The model parameters are updated according to the total loss value until the set training termination condition is met, wherein the total loss value includes at least the first loss value.

[0016] In one possible design, in another implementation of the first aspect of the embodiments of this application, the training process of the control parameter prediction model further includes:

[0017] Based on the image features, key control parameters for each row and column line are predicted. These key control parameters determine the starting position of the curve represented by the curve function.

[0018] Based on the key control parameters of each row and column obtained from the prediction, and the key control parameters contained in a set of control parameters corresponding to each row and column in the prediction results of the control parameters, the second loss value is calculated;

[0019] The total loss value also includes the second loss value.

[0020] In one possible design, in another implementation of the first aspect of the embodiments of this application, the training process of the control parameter prediction model further includes:

[0021] Based on the control parameter prediction results and the curve function, a first curve function for each row and column line is generated, and based on the first curve function, a first table structure corresponding to the control parameter prediction results is determined;

[0022] Based on the control parameter labels and the curve functions, a second curve function is generated for each row and column line, and the second table structure corresponding to the control parameter labels is determined according to the second curve functions;

[0023] Based on the degree of similarity between the first table structure and the second table structure, the structural similarity loss value is calculated;

[0024] The total loss value also includes the structural similarity loss value.

[0025] In one possible design, in another implementation of the first aspect of the embodiments of this application, the curve function is a Bézier curve function, the set of control parameters is a set of control points in the Bézier curve function, and the key control parameter is the first control point in the set of control points.

[0026] In one possible design, another implementation of the first aspect of the embodiments of this application further includes:

[0027] Based on the adjacency relationship between the smallest cell in the table image and the structured table, a first adjacency graph of the table image and a second adjacency graph of the structured table are generated respectively.

[0028] By comparing the adjacency differences between the smallest cells in the first adjacency graph and the second adjacency graph, a cell merging matrix is ​​generated, where each value in the cell merging matrix represents the merging direction of the smallest cell at its corresponding position.

[0029] Based on the cell merging matrix, the smallest cells in the structured table are merged to obtain the target structured cells.

[0030] In one possible design, in another implementation of the first aspect of the embodiments of this application, the step of inputting the table image into a configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image includes:

[0031] Extract the image features of the table image;

[0032] Attention information for the image features is calculated using an attention mechanism;

[0033] Based on the attention information, a set of control parameters corresponding to each row and column line in the table image is obtained.

[0034] In a second aspect, an electronic device is provided, comprising: a memory and a processor;

[0035] The memory is used to store programs;

[0036] The processor is configured to execute the program to implement the various steps of the table structuring method described in any of the preceding first aspects of this application.

[0037] Thirdly, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the table structuring method described in any of the preceding first aspects of this application.

[0038] Fourthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the various steps of the table structuring method described in any of the first aspects of this application.

[0039] Using the above technical solution, the table structuring method provided in this application predefines a curve function, in which the control parameters can flexibly adjust the shape and position of the curve, adapting to changes in different row and column lines, thus providing a flexible mathematical model for describing the row and column lines in a table. Based on this, the table image to be structured is input into a pre-trained control parameter prediction model. This model directly learns the mapping relationship between the table image and the control parameters of the curve function of the row and column lines in the table image, and can accurately estimate the control parameters controlling the shape and position of the curve function by identifying image features in the table image.

[0040] These control parameters directly define the curve function, meaning the model's output directly defines a curve, essentially predicting the function representation of each row and column line. Compared to existing technologies, this application omits the fitting process, thus eliminating the additional errors introduced by this process. This improves the accuracy of the structured table ultimately generated by the curve function of the row and column lines, further facilitating subsequent extraction, analysis, and processing of the table data. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 A schematic diagram of an implementation system architecture for the table structuring method provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating a table structuring method provided in an embodiment of this application;

[0044] Figure 3 Example diagram of control points for Bézier curves provided in embodiments of this application;

[0045] Figure 4 A flowchart illustrating the training of the control parameter prediction model provided in this application embodiment;

[0046] Figure 5 This example illustrates a schematic diagram of a training framework for a control parameter prediction model.

[0047] Figure 6 This example illustrates another training framework for a control parameter prediction model.

[0048] Figure 7 An example diagram of cell merging provided in this application embodiment;

[0049] Figure 8 An example diagram illustrating the application of a table structuring method provided in this application embodiment;

[0050] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In table-based structured scenarios, relevant technologies directly extract key points for row and column lines from the table image. The mathematical representation of these lines is then fitted to these key points to achieve table structuring. Here, the key points for row and column lines refer to the actual points in the table image, i.e., pixels. Therefore, the accuracy of key point detection depends on the image quality of the table image. In cases of low image quality, factors such as image noise, lighting variations, and blurred table lines can affect the accuracy of the extracted key point locations. Furthermore, fitting key points is a complex post-processing step, and the fitting algorithm itself introduces fitting errors. The accumulation and amplification of these fitting errors and key point detection errors result in low accuracy of the final structured table recognition.

[0053] To address the aforementioned problems, embodiments of this application provide a table structuring method to improve the accuracy of structured tables identified from table images. The table structuring method of this application will be described in detail below with reference to the accompanying drawings.

[0054] The method provided in this application can be applied to... Figure 1 In the system architecture shown, for example, the table structuring method is deployed in the cloud or server 200, and terminal 100 is the object that publishes table structuring tasks to the cloud or server 200. The implementation form of terminal 100 can be, but is not limited to, mobile phones, tablets, learning machines, teaching screens, robots, or wearable devices.

[0055] It is understood that either terminal 100 or server 200 can be used independently to execute the table structuring method provided in the embodiments of this application. In addition, terminal 100 and server 200 can also be used collaboratively to execute the table structuring method provided in the embodiments of this application.

[0056] For ease of understanding, combined with Figure 2 This application provides a flowchart of a table structuring method, and illustrates the method by applying it to a computer device, which can specifically be... Figure 1 Terminal 100 or server 200. (Refer to...) Figure 2 The table structuring method specifically includes the following steps:

[0057] Step S100: Obtain the table image to be structured and a preset curve function containing a set of unknown control parameters.

[0058] This step first obtains an image containing the table to be structured. This table image can be obtained from various data sources, such as scanned documents, image files, and live camera footage.

[0059] The preset curve function describes the shape and position of each row and column line in the table. A set of unknown control parameters within the curve function controls the shape and position of the curve it represents. For example, the curve function might be a quadratic function y(x) = ax². 2 If a, b, and c are a set of control parameters, then by adjusting the values ​​of a, b, and c, the shape and position of the curve represented by y(x) can be controlled. This can be understood as a method for drawing matrix lines designed by inversely finding a set of equation coefficients based on the known curve parametric equation.

[0060] It is understandable that the quadratic function in this embodiment is one possible implementation of a curve function. To flexibly adapt to changes in the rows and columns of the table, higher-order functions can be selected, or more complex function forms such as spline curves can be considered to define the curve function, thereby improving the accuracy and flexibility of the curve function in describing rows and columns. Furthermore, it should be noted that a straight line is a special case of a curve, and this curve function can also represent straight rows and columns.

[0061] In one possible implementation, the curve function can be a Bézier curve function. A Bézier curve function can fit a smooth curve using a set of control points. Therefore, in this embodiment, the set of control parameters in the curve function is the same as the set of control points in the Bézier curve function. Based on this, this embodiment can be understood as a row and column line drawing method designed by inversely deriving a set of control points from a known curve parametric equation.

[0062] Optionally, the Bézier curve can be a third-order Bézier curve function as shown in equation (1). The third-order Bézier curve function is defined by a set of control points, namely four control points P0, P1, P2, and P3, and the parameter t∈[0,1] is used to control the curve generation.

[0063] (1)

[0064] Where P0 and P3 represent the start and end points of the Bézier curve, respectively, and P1 and P2 represent the intermediate control vertices of the Bézier curve. (Refer to...) Figure 3 The example diagram of control points for a Bézier curve provided in this application illustrates the control points mentioned above. Points P0, P1, P2, and P3 define a cubic Bézier curve in a plane or three-dimensional space. The curve starts at P0, moves towards P1, and then from the direction of P2 to P3. However, the curve generally does not pass through P1 or P2; these two points are typically used to provide directional information for the curve. The distance between P0 and P1 determines the length of the curve moving towards P2 before turning towards P3.

[0065] It is understandable that all points on a straight line have the same orientation. Therefore, when a Bézier curve uses a set of control points to describe a straight line, all control points must be strictly collinear, meaning they all lie on the same straight line and are usually evenly distributed along that line to accurately describe the line. For example, when a third-order Bézier curve uses four control points P0, P1, P2, and P3 to describe a straight line segment, P0 and P3 are the start and end points of the segment, and P1 and P2 are the trisection points of the segment.

[0066] The following considerations were taken into account when choosing Bézier curves as the predefined curve function to draw row and column lines in this embodiment:

[0067] 1. Ensure the smoothness of row and column lines: Compared with a single line connecting data points, the curve fitted by Bézier curve is smooth and closer to the presentation of table row and column lines, making the final row and column line table closer to the original table.

[0068] 2. Significantly improved computational efficiency: In existing technologies, fitting a curve requires extracting a large number of key points (the points to be fitted). As the number of key points increases, a higher-order polynomial needs to be constructed to achieve the fitting. In contrast, Bézier curves require only a few control points (the control points are different from the key points to be fitted). For example, a third-order Bézier curve only needs four control points to fit complex line shapes, greatly reducing computational overhead and correspondingly improving the efficiency of table structuring.

[0069] Step S110: Input the table image into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image.

[0070] The pre-configured control parameter prediction model is trained on sample data of table images labeled with a set of control parameters corresponding to each row and column line. In this embodiment, the control parameter prediction model can be implemented using various types of models, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variants (e.g., LSTM, GRU). Furthermore, the control parameter prediction model learns from a large amount of labeled table image sample data, establishing a mapping relationship between the image features of the table and the control parameters describing the table's rows and columns, enabling it to predict the control parameters of the curve function corresponding to each row and column line in the input table image. It should be noted that in this application, "row and column line" refers to either a row line or a column line, and this limitation will not be repeated below.

[0071] Based on this, the table image containing the table to be structured, obtained in step S100, is input into the control parameter prediction model for prediction processing. This yields a set of control parameters for each row and column line of the table to be structured, output by the control parameter prediction model. This set of control parameters, combined with the curve function, can completely describe the shape and position of the corresponding row and column lines.

[0072] Step S120: Based on a set of control parameters and curve functions corresponding to each row and column line, generate the curve function corresponding to each row and column line.

[0073] Based on the set of control parameters corresponding to each row and column line obtained in step S110, and by updating a set of unknown control parameters in the preset curve function using these predicted control parameters, the curve function expression corresponding to each row and column line can be determined. For example, in step S100, the quadratic function y(x)=ax... 2 Taking +bx+c as an example, step S110 predicts a set of control parameters for the first line: a=0, b=3, c=1. Using the known a, b, and c, y(x)=ax is updated. 2 If a, b, and c are unknown in +bx+c, then the curve function expression y(x)=3x+1 of the line can be determined.

[0074] Step S130: Generate a structured table of the table image based on the curve function corresponding to each row and column line.

[0075] Optionally, on a blank canvas, each row and column line is drawn according to a curve function. The drawn row and column lines intersect each other, resulting in a visualized table structure rendered according to the curve function. Furthermore, the intersection points between all row and column lines can be calculated and treated as vertices of the table. The entire table is traversed, and the area enclosed by four adjacent vertices is defined as a cell. Logical coordinates are assigned to each cell, such as (2,3), representing the cell in the third column of the second row. The table content of each cell identified from the table image is associated with the corresponding logical coordinate cell to achieve the effect of filling the corresponding cell with the table content, resulting in a complete structured table. Finally, the structured table is stored according to certain rules and formats, such as JSON or XML, no longer containing any pixel information, but only pure logical information.

[0076] The core of this step is to transform the curve function of each row and column line obtained from S120 into a machine-readable, high-definition rendering structured data object.

[0077] The table structuring method provided in this embodiment differs from existing technologies. Existing technologies treat each row and column line in a table as a line connecting multiple key points, indirectly describing the row and column lines by fitting these key points. This embodiment treats each row and column line in the table as a curve function. A control parameter prediction model identifies image features in the table image to estimate the control parameters that determine the shape and position of the curve function. These control parameters directly define the curve function; that is, the model's output directly defines a curve, equivalent to the model directly predicting the function representation of each row and column line. Compared to existing technologies, this application omits the complex fitting process, improving the efficiency of table structuring while eliminating the additional errors introduced by the fitting process. This improves the accuracy of the structured table ultimately generated by the curve function of the row and column lines, further facilitating subsequent extraction, analysis, and processing of table data.

[0078] Furthermore, in existing line modeling techniques, the connections between key points may not be smooth, easily resulting in jagged lines. The continuity and smoothness of the final fitted matrix lines depend on the post-processing fitting algorithm. In contrast, this embodiment directly defines the matrix lines as a curve function used to describe smooth curves, resulting in naturally continuous and smooth lines.

[0079] Next, other possible implementations of the table structuring method provided in this application will be described through the following embodiments.

[0080] Reference Figure 4 The process of training the control parameter prediction model in this application embodiment includes the following steps:

[0081] Step S200: Obtain the table image sample and its corresponding control parameter label.

[0082] The main purpose of this step is to collect a dataset for training the model. The dataset includes table image samples and corresponding control parameter labels. The table image samples are image samples containing tables, which are the objects the model needs to learn to process. The control parameter labels are a set of control parameters corresponding to each row and column line obtained after converting the table row and column lines in the table image samples into curve functions. These are the expected output results of these table image samples, providing supervised information for model training, enabling the model to learn the mapping relationship from the table image to a set of control parameters for the row and column lines in the table image.

[0083] The table image samples can be collected through various means, such as scanning spreadsheet images from paper documents or directly extracting table images from electronic documents. The control parameter labels corresponding to the table image samples can be obtained through manual annotation. However, considering the diversity of images in the table image samples, manual annotation is costly and may not guarantee the consistency of control parameters for the same table image samples. Therefore, in some possible implementations, this application allows computer equipment to annotate the table image samples according to predefined annotation rules, ensuring that the annotation results for the table image samples are as consistent as possible.

[0084] Specifically, taking the process of labeling a row and column line in a table image sample with a set of control parameters as an example, the process of obtaining a table image sample and its corresponding control parameter labels is explained. The process involves detecting the table image sample, identifying each row and column line, and referring to... Figure 3 The solid line represents a row in the identified table, and one or more coordinate points P located on that row are taken. i (x i ,y i ). (P) i (x i ,y i Substituting into a predefined curve function, such as a third-order Bézier curve, yields the following equation (2), which contains a set of unknown control parameters P0, P1, P2, and P3. To solve for the four unknown control parameters, four coordinate points can be taken from the curve to construct a system of equations and solve for P0, P1, P2, and P3. It is understandable that different orders of Bézier curve functions or different selected curve functions contain different numbers of unknown control parameters, and therefore the number of known coordinate points selected will also be different. In actual annotation, the corresponding number of coordinate points can be selected according to the number of unknown control parameters in the selected curve function, which will not be listed here.

[0085] (2)

[0086] Furthermore, by solving the equation containing a set of unknown control parameters P0, P1, P2, and P3, we obtain the equation describing the coordinate point P. i (x i ,y i The control parameters P0, P1, P2, and P3 of the row and column lines are defined. Based on the above method, a set of control parameters corresponding to each row and column line is calculated, and the table image sample is labeled with a set of control parameters corresponding to each row and column line in the table image sample.

[0087] Step S210: Extract image features from the table image samples, and predict the control parameters based on the image features.

[0088] This step is the core processing part of the control parameter prediction model. First, meaningful image features are extracted from the table image samples. These image features can capture key information such as the table's structure, layout, and text information. Then, the predicted control parameters are obtained based on the extracted image features.

[0089] Reference Figure 5 The present application provides a schematic diagram of a control parameter prediction model training framework. The control parameter prediction model training process involves at least the following modules: feature extraction module and control parameter prediction module.

[0090] The feature extraction module is used to extract features from the table image samples to obtain image features. The extracted image features are used as input to the control parameter prediction module.

[0091] The feature extraction module can use convolutional neural networks (CNNs) from deep learning to extract features. For example, pre-trained models such as VGG and ResNet can be used as feature extractors. The table image samples are input into these models, and the output of the last or intermediate layers is obtained as image features. Alternatively, a dedicated feature extraction network can be designed based on the characteristics of the table image, such as extracting features from the lines, cells, and other structural elements of the table.

[0092] In one possible implementation, this embodiment uses a convolutional neural network (such as ResNet50, Swin Transformer, etc.) pre-trained on the ImageNet dataset on a large scale as a visual feature extractor to construct the feature extraction module. Leveraging its general visual representation capabilities learned from massive amounts of natural images, it can efficiently map the input tabular image onto a high-dimensional feature space, thereby capturing key image information such as basic texture, edge contours, and semantic structure.

[0093] This embodiment aims to further explore the multi-level and multi-scale image features of table images, fully utilizing the differentiated representation capabilities of different network depth layers for image features. Specifically, the outputs of the first four layers are selected as image features, denoted as F1, F2, F3, and F4. The shallower layers F1 and F2 focus on extracting low-level detail features of the table image, such as line textures and local edges; while the deeper layers F3 and F4 tend to capture higher-level semantic features, such as the overall layout of the table and cell distribution patterns.

[0094] Furthermore, considering the differences in resolution and number of channels in the feature maps output from different layers, sampling techniques such as bilinear interpolation and max pooling are employed. Features with the same adjustment value and resolution are then concatenated along the channel dimension to achieve effective feature fusion, resulting in a fused feature F. This fused feature F is then used for the next step of control parameter prediction. This multi-scale feature fusion strategy integrates multi-level information from the bottom to the top layers, preserving the fine structural information of the table image while fusing global semantic layout features to improve the accuracy of feature representation and further optimize the accuracy of the prediction results.

[0095] In another possible implementation, the table image is input into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image, including: extracting image features of the table image; calculating attention information on the image features through an attention mechanism; and obtaining a set of control parameters corresponding to each row and column line in the table image based on the attention information.

[0096] Specifically, the aforementioned feature extraction network can be used as the backbone network of the feature extraction module to perform multi-scale feature extraction on the table image, capturing visual information including lines and overall layout. Simultaneously, the feature extraction module integrates an attention mechanism network. The function of the attention mechanism network is to determine the key regions of the table rows and columns that should be focused on at the current feature extraction moment, enhancing the model's ability to perceive the table rows and columns and their curvature details.

[0097] The image features extracted by the backbone network are further processed through an attention mechanism network. The resulting attention information mainly includes row and column features from the image features. Furthermore, a set of control parameters corresponding to the row and column features is predicted based on these row and column features.

[0098] It is understandable that the training phase described in step S210, in which the control parameter prediction model predicts table image samples, is also applicable to the inference phase, which is the phase in which the trained control parameter prediction model is used to predict table images.

[0099] Step S220: Calculate the first loss value based on the prediction results of the control parameters and the control parameter labels.

[0100] Step S230: Update the model parameters according to the total loss value until the set training termination condition is met. The total loss value includes at least the first loss value L1.

[0101] Optionally, the mean squared error between the predicted control parameters and the control parameter labels is calculated as the first loss value L1 to measure the difference between the predicted results of the control parameter prediction model and the true labels. An optimization algorithm is then used to update the model's parameters, making the model's predictions closer to the true labels. This process is iterated until the set training termination conditions are met, such as reaching the maximum number of iterations or the total loss value converging to a certain range, thus obtaining a control parameter prediction model with better performance.

[0102] The training process of the aforementioned control parameter training model, by collecting and utilizing a large number of table image samples and their corresponding labels, can fully leverage the information in the data to uncover the potential patterns between the table images and the control parameters, thereby obtaining more accurate prediction results. Furthermore, by calculating the loss value and updating the model parameters, the model can continuously learn and improve, gradually enhancing its understanding and prediction capabilities regarding table images. As training progresses, the model's performance will continuously improve, better adapting to different table image and control parameter requirements.

[0103] Furthermore, the control parameter prediction model accurately estimates the control parameters that control the shape and position of the curve function by identifying image features in the table image, and determines the specific curve function representing the row and column lines based on the control parameters output by the model. Compared with existing methods that extract pixel key points to fit row and column lines, this application estimates the control parameters by identifying overall features in the image through the model, rather than relying solely on local key points in the image, which greatly reduces the recognition error caused by factors such as image noise.

[0104] In one possible implementation, the training process of the control parameter prediction model may further include: predicting the key control parameters of each row and column line based on image features, wherein the key control parameters are the control parameters that determine the starting position of the curve represented by the curve function; calculating the second loss value L2 based on the predicted key control parameters of each row and column line and the key control parameters contained in a set of control parameters corresponding to each row and column line in the control parameter prediction results; the total loss value also includes the second loss value L2.

[0105] Reference Figure 6 This application provides a schematic diagram of another control parameter prediction model training framework, which details the training process of the control parameter prediction model.

[0106] The image features extracted by the feature extraction module are used as input to the key control parameter prediction module. This module predicts the key control parameters for each row and column line of the table image sample. The key control parameters determine the starting position of the curve represented by the curve function. When the curve function is a Bézier curve, the key control parameter is the first control point in a set of control points, also known as the starting point P0.

[0107] It is understandable that key control parameters (or starting points) are control parameters that determine the starting position of the curve represented by the curve function, and to a certain extent constrain the starting direction and shape of the curve. Therefore, this embodiment jointly trains the key control parameter prediction module and the control parameter prediction model (feature extraction module and control parameter prediction module). During the joint training process, the prediction results of the key control parameter prediction model are used as a supervision signal for the control parameter prediction results, so as to provide more accurate positioning parameters and optimize the performance of the control parameter prediction model.

[0108] In some scenarios, the row and column lines in a table image may overlap due to factors such as shooting angle and device resolution, making it difficult for the control parameter prediction model to accurately identify and distinguish these overlapping lines. To address this, a second loss value is used, calculated by comparing the number of control parameter groups in the prediction results with the total number of critical control parameters in the critical control parameter prediction model. This allows the control parameter prediction model to learn how to correctly identify row and column lines in such complex situations. Even with overlapping lines, it can accurately identify the control parameter group corresponding to each line, reducing missed detections and significantly improving the model's adaptability to complex image scenes.

[0109] In another scenario, the control parameter prediction results contain multiple sets of control parameters, with each set corresponding to a row and column line. The number of control parameter sets should be equal to the total number of rows and columns in the table, that is, equal to the total number of key control parameters. However, the rows and columns in the table image are affected by noise, interference lines, and other factors, resulting in blurred rows and columns. This leads to a problem where, even if the number of predicted control parameter sets is the same as the total number of rows and columns, the model still has ambiguity in locating the starting point of the rows and columns, i.e., the key control parameter prediction is inaccurate.

[0110] To address this, by matching the key control parameters in each group of control parameters in the control parameter prediction results with the key control parameters predicted by the key control parameter prediction module, the key control parameters of each row and column can be accurately located. Furthermore, the distance deviation between the two closely matched key control parameters is calculated as a second loss value, allowing the control parameter prediction model to more precisely measure the difference between the predicted key control parameters (those in the control parameter prediction results) and the reference key control parameters (those predicted by the key control parameter prediction module). Introducing this second loss value during training forces the control parameter prediction model to pay closer attention to the key control parameters of each row and column, reducing ambiguity in the location and enabling the model to accurately identify the key control parameters of rows and columns even under interference, thus significantly improving the location accuracy of rows and columns.

[0111] The training process of the control parameter prediction model provided in this embodiment, by training the control parameter prediction model (feature extraction module and control parameter prediction module) together with the key control parameter prediction module during the training phase, that is, by adopting a multi-task training strategy (control parameter prediction task and key control parameter prediction task), can bring significant advantages to the control parameter prediction model finally used for inference.

[0112] In summary, the control parameter prediction model training process provided in the above embodiments of this application utilizes the key control parameter prediction module's task of predicting key control parameters for table image samples as a powerful supervisory signal and guide to enhance the intrinsic capabilities of the control parameter prediction model. Finally, even if the control parameter prediction model is removed from the key control parameter prediction module during the inference phase, the benefits of its training are already embedded in the retained control parameter prediction model, resulting in a final lightweight control parameter prediction model that significantly outperforms the model trained solely with control parameter labels.

[0113] In another possible implementation, the training process of the control parameter prediction model may further include: generating a first curve function for each row and column based on the control parameter prediction results and the curve function, and determining a first table structure corresponding to the control parameter prediction results based on the first curve function; generating a second curve function for each row and column based on the control parameter labels and the curve function, and determining a second table structure corresponding to the control parameter labels based on the second curve function; calculating a structural similarity loss value based on the similarity between the first table structure and the second table structure; the total loss value also includes the structural similarity loss value.

[0114] The core of this training process is to supervise the model not only at the parameter level of the curve function, but also at the structural level.

[0115] Specifically, firstly, using the control parameter prediction results from the control parameter prediction model, a row and column line is rendered using a predefined curve function (such as a Bézier curve), forming the "first curve function." This essentially transforms the abstract control parameter values ​​into concrete geometric shapes. Further, the rendered row and column lines (the "first curve function") are combined to reconstruct a complete table structure, which can be denoted as the "first table structure." This table structure clearly defines the position, shape, and adjacency relationships of all cells. Similarly, for multiple sets of control parameters and the same curve function in the control parameter labels, a "second table structure" is rendered and reconstructed.

[0116] From the perspective of overall topology and geometry, the similarity between the two reconstructed "first table structure" and "second table structure" is calculated as the structural similarity loss value. This structural similarity loss value, together with the previous first loss value and / or second loss value, constitutes the total loss, which is used to update the model parameters of the control parameter prediction model.

[0117] Introducing structural similarity loss values ​​for control parameter prediction models allows them to move from local parameter optimization to global structural optimization, bringing significant advantages to the final control parameter prediction models used for inference. These advantages are specifically reflected in the following aspects:

[0118] 1. Improve the overall accuracy of the table structure. The first loss value L1 and the second loss value L2 can only ensure that the predicted parameters of a single line are close to the label value. However, even if the error of each line is small, the accumulation of small parameter errors can lead to serious structural errors. For example, two lines that should be connected may fail to connect due to a slight deviation in the prediction of the endpoints, resulting in a cell being incorrectly split into two, or two cells being incorrectly merged into one.

[0119] The structural similarity loss value directly targets whether the final output table structure is correct. It can keenly capture topological errors and severe geometric deviations, and feed them back to the model through the loss function, forcing the model to learn how to produce structurally correct predictions, rather than just predictions that are numerically similar at the level of individual lines. This greatly improves the usability and reliability of the final analysis results.

[0120] 2. Enhancing the robustness and generalization ability of the model. Different table images may correspond to tables with different parameter values ​​but identical topological structures. For example, after a table is translated, scaled, or slightly rotated, the values ​​of all its control parameters change, but the table structure remains the same. Models that simply use parametric losses (L1, L2) may treat these as two completely different samples, thus reducing generalization ability. Structural similarity loss, on the other hand, focuses more on the similarity of structures rather than the absolute values ​​of parameters. It guides the model to learn the concept of "structural equivalence," making the model naturally robust to such geometric transformations and better able to generalize to tables of different styles, sizes, and orientations.

[0121] In summary, this embodiment elevates the supervision signal from low-level numerical regression to high-level semantic structural understanding by introducing a "structural similarity loss value." This enables the model to learn the high-level semantic features of the table constructed by rows and columns in a table image. The learning objective directly aligns with the final task of table structuring, namely, generating a correct table grid structure. Compared to existing technologies, the model can semantically determine the true shape of the table when faced with interference such as image noise, reducing the impact of local parameter changes caused by noise. This significantly improves the overall accuracy, robustness, and generalization of the parsing results.

[0122] Furthermore, the control parameter prediction model trained through the above training process is applied to the inference stage of steps S100-S130 to obtain a set of control parameter prediction results for the row and column lines of the table. By performing precise pairwise intersection calculations on the row and column lines, the basic grid structure of the table is constructed, resulting in a structured table.

[0123] In one possible implementation, to reconstruct cells that were incorrectly split due to the intersection of row and column lines, after obtaining the structured table in step S130, the method further includes: generating a first adjacency graph of the table image and a second adjacency graph of the structured table based on the adjacency relationships between the smallest cells in the table image and the structured table, respectively; comparing the differences in adjacency relationships between the smallest cells in the first adjacency graph and the second adjacency graph to generate a cell merging matrix, where each value in the cell merging matrix represents the merging direction of the smallest cell at its corresponding position; and merging the smallest cells in the structured table according to the cell merging matrix to obtain the target structured cell.

[0124] Reference Figure 7This application provides an example diagram of cell merging. During the structuring process of the table image, the intersection of each row and column line predicted by the model can lead to the incorrect splitting of cells, such as Cell2 being incorrectly split into two smallest cells. To address this, this embodiment merges the incorrectly split cells in the structured table by utilizing the adjacency relationship between the structured table and the cells in the table image, resulting in a target structured cell that matches the table image.

[0125] Specifically, first, a first adjacency graph of the table image is constructed, and a second adjacency graph of the structured table is constructed. (See reference...) Figure 7 The first adjacency graph in the table is a graphical tool that describes the topological relationship between the smallest cells in the table. It uses the connection relationship between "nodes" and "edges". A "node" represents the smallest cell, and the line connecting two nodes, i.e., the "edge", represents the adjacency relationship between the two smallest cells corresponding to the two nodes.

[0126] Subsequently, by comparing the first and second adjacency graphs, the actual adjacency relationship between each cell in the structured table and its adjacent cells is calculated. For example, Cell1-1 and Cell1-2 correspond to Cell1, and logically belong to the same cell, both adjacent to Cell2-1 and Cell3; Cell2-1 and Cell2-2 correspond to Cell2, and logically belong to the same cell, both adjacent to Cell3 and Cell4; Cell3 and Cell4 are independent. Based on this, the cell merging matrix of the structured table is generated, resulting in a 2×3 matrix as shown in equation (3). Each value in the cell merging matrix represents the merging direction of the smallest cell at its corresponding position, such as "1" representing merging the cell to the left, "2" representing merging the cell upwards, and "0" representing no merging.

[0127] (3)

[0128] Furthermore, the cell merging matrix is ​​traversed, identifying whether the values ​​of the cells to the right and below the current cell meet the merging condition, i.e., whether they are "1" or "2". If so, the cells are merged to the right or down. Alternatively, only the value of the current cell is checked to see if it meets the merging condition, i.e., whether the value of the current cell is "1" or "2". If so, the cells are merged to the left or up; otherwise, the cells remain independent. Based on this, each smallest cell in the structured table is merged to obtain a target structured table with the same cell adjacency relationship as the table image, restoring the logical structural integrity of the table.

[0129] Reference Figure 8The present application provides an example diagram of a table structuring method, which uses a Bézier curve as a preset curve function to train a control parameter prediction model and to illustrate the table structuring process.

[0130] The training phase of the control parameter prediction model is illustrated with an example. The control parameter prediction model includes a feature extraction network, a row line prediction module, and a column line prediction module. The feature extraction network extracts image features from the input table image; the row line prediction module predicts a set of control points corresponding to each row line based on the image features; and the column line prediction module predicts a set of control points corresponding to each column line based on the image features.

[0131] During the training phase, the row line prediction module and the row line start point prediction module are trained together, and the column line prediction module and the column line start point prediction module are trained together. During the joint training process, the predicted row line start points and column line start points are used to supervise the learning of the output results of the row line prediction module and the column line prediction module respectively.

[0132] It is understandable that the feature extraction network is equivalent to the feature extraction module mentioned above, and the row line prediction module and column line prediction module are equivalent to the control parameter prediction module mentioned above. The feature extraction network, row line prediction module, and column line prediction module together form the control parameter prediction model, and the row line start point prediction module and column line start point prediction model are equivalent to the key control parameter prediction module mentioned above. The training of the control parameter prediction model in this example, and the process of jointly training the control parameter prediction model with the row line start point prediction module and column line start point prediction module, can be referred to the description above and will not be repeated here.

[0133] The inference phase of the control parameter prediction model trained above is illustrated with an example. The table image to be structured is input into the control parameter prediction model. The feature extraction network extracts image features from the table image. These image features are then used for prediction by the row and column line prediction modules, resulting in a set of control points corresponding to each row and column line. Based on these control points, a pre-defined Bézier curve function formula is substituted to obtain the Bézier curves corresponding to each row and column line. Finally, the table is plotted based on these Bézier curves, resulting in a structured table.

[0134] Furthermore, adjacency graphs are constructed for the corresponding cells in the table image and the structured table. Then, based on these adjacency graphs, the topological adjacency relationship between each cell and its neighboring cells is calculated, generating a merge decision matrix for the structured table. The merge decision matrix is ​​traversed. For each cell, if the cell below it meets the merge condition (e.g., the corresponding value of the cell below it in the merge decision matrix is ​​"2"), it is merged downwards. If the cell to its right meets the merge condition (e.g., the corresponding value of the cell to its right in the merge decision matrix is ​​"1"), it is merged to the right. If neither condition is met, the cell is considered an independent cell and remains independent. Finally, the merge operation is recursively executed based on the merge decision matrix to restore the logical structural integrity of the table.

[0135] This application also provides an electronic device in its embodiments. (See reference...) Figure 9 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, terminals such as mobile phones, tablets, learning machines, large-screen teaching displays, robots, wearable devices, etc. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0136] like Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the table structuring method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0137] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0138] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the table structuring methods provided in this application.

[0139] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the table structuring methods provided in this application.

[0140] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0143] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0144] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A table structuring method, characterized in that, include: Obtain the table image to be structured and a preset curve function containing a set of unknown control parameters, wherein the control parameters are used to control the shape and position of the curve represented by the curve function; The table image is input into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image. The control parameter prediction model is trained using table image sample data labeled with a set of control parameters corresponding to each row and column line. Based on a set of control parameters corresponding to each row and column line and the curve function, generate the curve function corresponding to each row and column line. A structured table of the table image is generated based on the curve function corresponding to each row and column line.

2. The table structuring method according to claim 1, characterized in that, The training process of the control parameter prediction model includes: Obtain a table image sample and its corresponding control parameter labels, wherein the control parameter labels include a set of control parameters corresponding to each row and column line in the table image sample; Extract image features from the table image samples, and predict control parameters based on the image features; Calculate the first loss value based on the prediction results of the control parameters and the control parameter labels; The model parameters are updated according to the total loss value until the set training termination condition is met, wherein the total loss value includes at least the first loss value.

3. The table structuring method according to claim 2, characterized in that, The training process of the control parameter prediction model also includes: Based on the image features, key control parameters for each row and column line are predicted. These key control parameters determine the starting position of the curve represented by the curve function. Based on the key control parameters of each row and column obtained from the prediction, and the key control parameters contained in a set of control parameters corresponding to each row and column in the prediction results of the control parameters, the second loss value is calculated; The total loss value also includes the second loss value.

4. The table structuring method according to claim 2, characterized in that, The training process of the control parameter prediction model also includes: Based on the control parameter prediction results and the curve function, a first curve function for each row and column line is generated, and based on the first curve function, a first table structure corresponding to the control parameter prediction results is determined; Based on the control parameter labels and the curve functions, a second curve function is generated for each row and column line, and the second table structure corresponding to the control parameter labels is determined based on the second curve functions. Based on the degree of similarity between the first table structure and the second table structure, the structural similarity loss value is calculated; The total loss value also includes the structural similarity loss value.

5. The table structuring method according to any one of claims 1-4, characterized in that, The curve function is a Bézier curve function, the set of control parameters is a set of control points in the Bézier curve function, and the key control parameter is the first control point in the set of control points.

6. The table structuring method according to any one of claims 1-4, characterized in that, Also includes: Based on the adjacency relationship between the smallest cell in the table image and the structured table, a first adjacency graph of the table image and a second adjacency graph of the structured table are generated respectively. By comparing the adjacency differences between the smallest cells in the first adjacency graph and the second adjacency graph, a cell merging matrix is ​​generated, where each value in the cell merging matrix represents the merging direction of the smallest cell at its corresponding position. Based on the cell merging matrix, the smallest cells in the structured table are merged to obtain the target structured cells.

7. The table structuring method according to any one of claims 1-4, characterized in that, The step of inputting the table image into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image includes: Extract the image features of the table image; Attention information for the image features is calculated using an attention mechanism; Based on the attention information, a set of control parameters corresponding to each row and column line in the table image is obtained.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the table structuring method as described in any one of claims 1 to 7.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the table structuring method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the various steps of the table structuring method as described in any one of claims 1 to 7.

Images