A table structuring method, related device and program product

Abstract

The application discloses a table structuring method, related equipment and program products, and relates to the technical field of computer vision and table recognition. The application inputs a table image to be structured into a configured control parameter prediction model, obtains a group of control parameters corresponding to each row and column line in the table image, generates a curve function corresponding to each row and column line based on the group of control parameters corresponding to each row and column line and a preset curve function containing a group of unknown control parameters, and generates a structured table of the table image according to the curve function corresponding to each row and column line. The application directly predicts the function expression of each row and column line, eliminates the additional error caused by the fitting process, and thus improves the accuracy of the structured table generated by the curve function of the row and column line.

Description

Technical Field

[0001] This application relates to the fields of computer vision and table recognition technology, and more specifically, to a table structuring method, related equipment and program products. Background Technology

[0002] Tables, as an efficient way to combine and present data, are widely used in various documents, such as office documents, reports, and web pages. However, this tabular data often exists in unstructured or semi-structured forms, which is not conducive to efficient retrieval, analysis, and management by computers. Therefore, there is an urgent need for table structuring methods to transform this complex tabular data into unified, standardized structured data, enabling computers to process it conveniently.

[0003] CN119992578A discloses a table recognition method that extracts key points of row lines and column lines from a table image based on the row and column features, respectively. Row and column dividing lines are then fitted using these key points to generate a structured table. However, during the key point fitting process, the fitting algorithm introduces fitting errors, resulting in low accuracy of the final recognized structured table.

[0004] Therefore, there is an urgent need for a method to accurately identify tables in images in order to structure the tables. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a table structuring method, related equipment, and program products to improve the accuracy of table recognition. The specific solution is as follows:

[0006] Firstly, a table structuring method is provided, including:

[0007] Obtain the table image to be structured and a preset curve function containing a set of unknown control parameters, wherein the control parameters are used to control the shape and position of the curve represented by the curve function;

[0008] The table image is input into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image. The control parameter prediction model is trained using table image sample data labeled with a set of control parameters corresponding to each row and column line.

[0009] Based on a set of control parameters corresponding to each row and column line and the curve function, generate the curve function corresponding to each row and column line.

[0010] A structured table of the table image is generated based on the curve function corresponding to each row and column line.

[0011] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the training process of the control parameter prediction model comprises:

[0012] obtaining a table image sample and a corresponding control parameter label of the table image sample, the control parameter label comprising a set of control parameters corresponding to each row and column line in the table image sample;

[0013] extracting image features of the table image sample, and predicting a control parameter prediction result based on the image features;

[0014] calculating a first loss value based on the control parameter prediction result and the control parameter label;

[0015] updating model parameters according to a total loss value until a set training end condition is reached, the total loss value comprising at least the first loss value.

[0016] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the training process of the control parameter prediction model further comprises:

[0017] predicting a key control parameter of each row and column line based on the image features, the key control parameter being a control parameter that determines a starting position of a curve represented by the curve function;

[0018] calculating a second loss value according to the predicted key control parameter of each row and column line and a key control parameter contained in a set of control parameters corresponding to each row and column line in the control parameter prediction result;

[0019] The total loss value further comprises the second loss value.

[0020] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the training process of the control parameter prediction model further comprises:

[0021] generating a first curve function of each row and column line based on the control parameter prediction result and the curve function, and determining a first table structure corresponding to the control parameter prediction result according to the first curve function;

[0022] generating a second curve function of each row and column line based on the control parameter label and the curve function, and determining a second table structure corresponding to the control parameter label according to the second curve function;

[0023] calculating a structure similarity loss value based on a similarity degree between the first table structure and the second table structure;

[0024] The total loss value further includes the structural similarity loss value.

[0025] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the curve function is a Bezier curve function, and the set of control parameters is a set of control points in the Bezier curve function, and the key control parameter is a first control point in the set of control points.

[0026] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the method further includes:

[0027] generating a first adjacency graph of the table image and a second adjacency graph of the structured table based on adjacency relationships between minimum cells in the table image and the structured table respectively;

[0028] generating a cell merging matrix by comparing adjacency relationship differences between each corresponding minimum cell in the first adjacency graph and the second adjacency graph, where each numerical value in the cell merging matrix represents a merging direction of a minimum cell at a corresponding position;

[0029] performing merging processing on the minimum cells in the structured table according to the cell merging matrix to obtain target structured cells.

[0030] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the inputting the table image into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image includes:

[0031] extracting image features of the table image;

[0032] calculating attention information on the image features through an attention mechanism;

[0033] obtaining a set of control parameters corresponding to each row and column line in the table image based on the attention information.

[0034] In a second aspect, an electronic device is provided, including a memory and a processor.

[0035] The memory is configured to store a program.

[0036] The processor is configured to execute the program to implement each step of the table structuring method described in any one of the preceding first aspects of the present application.

[0037] In a third aspect, a readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, each step of the table structuring method described in any one of the preceding first aspects of the present application is implemented.

[0038] In a fourth aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the table structuring method according to any one of the preceding first aspect.

[0039] By means of the above technical solution, the table structuring method provided by the present application pre-defines a curve function, control parameters of which can flexibly adjust the shape and position of the curve, and can adapt to the changes of different row and column lines, thereby providing a flexible mathematical model for describing the row and column lines in the table. On this basis, the table image to be structured is input into the pre-trained control parameter prediction model, wherein the control parameter prediction model directly learns the mapping relationship of the control parameters of the curve function of the row and column lines in the table image to the table image, and can accurately estimate the control parameters for controlling the shape and position of the curve function by recognizing the image features in the table image.

[0040] The control parameters can directly define the curve function, i.e., the output of the model directly defines a curve, which is equivalent to that the model directly predicts the function representation of each row and column line. Compared with the prior art, the present application omits the fitting process, thereby eliminating the additional error caused by the fitting process, and improving the accuracy of the structured table generated by the curve function of the row and column lines. Further, it is convenient for subsequent extraction, analysis and processing of table data, etc. BRIEF DESCRIPTION OF DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the present application. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0042] Figure 1 An embodiment system architecture schematic diagram of the table structuring method provided by the present application is shown in the following figure:

[0043] Figure 2 A flowchart of the table structuring method provided by the present application is shown in the following figure:

[0044] Figure 3 A control point example diagram of the Bezier curve provided by the present application is shown in the following figure:

[0045] Figure 4 A flowchart of training the control parameter prediction model provided by the present application is shown in the following figure:

[0046] Figure 5 An example of a control parameter prediction model training framework schematic diagram is shown in the following figure:

[0047] Figure 6 Another control parameter prediction model training framework schematic diagram is exemplified.

[0048] Figure 7 An example diagram of cell merging provided for an embodiment of the present application;

[0049] Figure 8 An application example diagram of a table structuring method provided for an embodiment of the present application;

[0050] Figure 9 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] In the table structuring scenario, the related art directly extracts the key points of the row and column lines from the table image, obtains the mathematical representation of the row and column lines through fitting of the key points, and realizes table structuring. The key points of the row and column lines refer to the real points in the table image, that is, the pixel points. Therefore, the detection accuracy of the key points depends on the image quality of the table image. In the case of low image quality, the extracted key points may be inaccurate due to the influence of various factors such as image noise, illumination change, and blurred table lines. Moreover, the process of fitting the key points is a complex post-processing process, and the fitting algorithm itself also produces fitting errors. The cumulative and amplification of the fitting errors and the key point detection errors result in low accuracy of the finally recognized structured table.

[0053] To solve the above problems, the table structuring method provided by the embodiments of the present application is used to improve the accuracy of the structured table recognized from the table image. The table structuring method of the embodiments of the present application will be described in detail below with reference to the drawings.

[0054] The method provided by the present application can be applied to Figure 1 The system architecture shown, for example, the table structuring method is deployed in the cloud or the server 200, and the terminal 100 is used as an object to publish a table structuring task to the cloud or the server 200, wherein the implementation form of the terminal 100 can be but is not limited to a mobile phone, a tablet computer, a learning machine, a teaching large screen, a robot, or a wearable device, etc.

[0055] It can be understood that the terminal 100 or the server 200 can be used alone to execute the table structuring method provided by the embodiments of the present application, and in addition, the terminal 100 and the server 200 can be used cooperatively to execute the table structuring method provided by the embodiments of the present application.

[0056] For the convenience of understanding, the flowchart of the table structuring method provided by the embodiments of the present application is combined with Figure 2 the table structuring method provided by the embodiments of the present application is applied to a computer device, which can be specifically the terminal 100 or the server 200 in Figure 1 . Referring to Figure 2 , the table structuring method specifically includes the following steps:

[0057] Step S100, acquiring a table image to be structured and a preset curve function containing a group of unknown control parameters.

[0058] This step first acquires an image containing a table to be structured. The table image can be acquired from various data sources, such as scanned documents, picture files, real-time shooting by a camera, and the like.

[0059] The preset curve function is used to describe the shape and position characteristics of each row and column line in the table. A group of unknown control parameters contained in the curve function is used to control the shape and position of the curve represented by the curve function. For example, the curve function is a monomial quadratic function y(x) = ax 2 +bx+c, and a, b, and c are a group of control parameters. By adjusting the values of a, b, and c, the shape and position of the curve represented by y(x) are controlled. It can be understood that the row and column line drawing method is designed according to the idea of inversely solving a group of equation coefficients from a known curve parameter equation.

[0060] It can be understood that the monomial quadratic function in this embodiment is a possible implementation of the curve function. In order to be able to flexibly adapt to the changes of the row and column lines in the table, a higher order function or a more complex function form such as a spline curve can also be selected to set the curve function, so as to improve the accuracy and flexibility of the curve function in describing the row and column lines. In addition, it should be noted that a straight line is a special case of a curve, and this curve function can also represent a straight row and column line.

[0061] In a possible implementation, the curve function can select a Bezier curve function. The Bezier curve function can fit a smooth curve through a group of control points. Therefore, in this embodiment, a group of control parameters in the curve function is a group of control points in the Bezier curve function. On this basis, this embodiment can be understood as a row and column line drawing method designed according to the idea of inversely solving a group of control points from a known curve parameter equation.

[0062] Optionally, the Bezier curve can use the third-order Bezier curve function shown in the following formula (1), which is defined by a set of control points, i.e., four control points P0, P1, P2, P3, and the parameter t∈[0,1] is used to control the curve generation.

[0063] (1)

[0064] Wherein, P0 and P3 represent the starting point and the ending point of the Bezier curve respectively, and P1 and P2 represent the intermediate control vertices of the Bezier curve respectively. Referring to Figure 3 , the control point example diagram of the Bezier curve provided by the embodiment of the present application is used to describe the above control points. The four points P0, P1, P2, P3 define a cubic Bezier curve in a plane or in a three-dimensional space. The curve starts from P0 and goes to P1, and comes to P3 from the direction of P2. However, the curve generally does not pass through P1 or P2, and these two points are usually used to provide the direction information of the curve. The distance between P0 and P1 determines how long the curve goes to the direction of P2 before turning to approach P3.

[0065] It can be understood that all the points on a straight line have the same direction, so when the Bezier curve describes a straight line using a set of control points, all the control points need to be strictly collinear, i.e., all the control points are located on the same straight line, and are usually uniformly distributed on the straight line, so as to accurately describe the straight line. For example, when the third-order Bezier curve describes a straight line segment using four control points P0, P1, P2, P3, P0 and P3 are the starting point and the ending point of the straight line segment, and P1 and P2 are the trisection points of the straight line segment.

[0066] The embodiment selects the Bezier curve as the predefined curve function for drawing the row and column lines, considering the following:

[0067] 1. Ensure the smoothness of the row and column lines: Compared with the polyline connecting the data points, the curve fitted by the Bezier curve is smooth, which is closer to the presentation form of the table row and column lines, so that the finally drawn table row and column lines are closer to the original table.

[0068] 2. The calculation efficiency is greatly improved: in the prior art, a large number of key points (points to be fitted) need to be extracted to fit a curve, and as the number of key points increases, a higher-order polynomial needs to be constructed to achieve fitting. Compared with the prior art, the Bezier curve only needs a small number of control points (control points are different from key points to be fitted), such as four control points for the third-order Bezier curve, which can fit complex line shapes, greatly reducing the calculation cost, and correspondingly improving the table structuring efficiency.

[0069] In step S110, the table image is input into the configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image.

[0070] The pre-configured control parameter prediction model is trained by using table image sample data labeled with a set of control parameters corresponding to each row and column. In the embodiments of the present application, the control parameter prediction model can be implemented by using various types of models, such as a convolutional neural network (CNN), a recurrent neural network (RNN), and variants thereof (such as LSTM and GRU). Further, the control parameter prediction model learns a large amount of labeled table image sample data, establishes a mapping relationship between the image features of the table and the control parameters describing the rows and columns of the table, and enables the control parameter prediction model to predict the control parameters of the curve function corresponding to each row and column in the table image according to the input table image. It should be noted that the row and column in the present application represent a row or a column, which will not be repeated in the following description.

[0071] On this basis, the table image containing the table to be structured obtained in step S100 is input into the control parameter prediction model for prediction processing, and a set of control parameters of each row and column of the table to be structured output by the control parameter prediction model is obtained. The set of control parameters can completely describe the shape and position of the corresponding row and column in combination with the curve function.

[0072] In step S120, a curve function corresponding to each row and column is generated based on a set of control parameters corresponding to each row and column and the curve function.

[0073] According to the set of control parameters corresponding to each row and column obtained in step S110, the set of unknown control parameters in the preset curve function is updated by using the set of predicted control parameters, and the curve function expression corresponding to each row and column is determined. Taking the quadratic function y(x) = ax 2 + bx + c in step S100 as an example, the set of control parameters of the first row obtained in step S110 is a = 0, b = 3, and c = 1. The unknown a, b, and c in y(x) = ax 2 + bx + c are updated by using the known a, b, and c, and the curve function expression y(x) = 3x + 1 of the row is determined.

[0074] In step S130, a structured table of the table image is generated according to the curve function corresponding to each row and column.

[0075] Optionally, on a blank canvas, each row and column line is drawn according to the curve function of each row and column line, and the drawn row lines and column lines intersect with each other to obtain a visual table structure rendered according to the curve function. Further, the intersection points between all the row and column lines can be calculated, and the intersection points are regarded as vertices of the table. The region surrounded by the adjacent four vertices is defined as a cell, and each cell is assigned a logical coordinate, such as (2, 3), representing the cell in the second row and the third column. The table content of each cell identified from the table image is associated with the cell corresponding to the logical coordinate, so as to fill the table content into the corresponding cell, and a complete structured table is obtained. Finally, the structured table is stored in a certain rule and format, such as JSON, XML, etc., and no longer contains any pixel information, but only logical information.

[0076] The core of this step is to convert the curve function of each row and column line obtained in S120 into a machine-readable and high-definition-renderable structured data object.

[0077] The table structuring method provided in this embodiment is different from the prior art. The prior art regards each row and column line in the table as a connection line of a plurality of key points, and indirectly describes the row and column line by fitting the key points on the line. This embodiment regards each row and column line in the table as a curve function, estimates and determines the control parameters of the shape and position of the curve function by identifying the image features in the table image through the control parameter prediction model, and the control parameters can directly define the curve function, that is, the output of the model directly defines a curve, which is equivalent to that the model directly predicts the function representation of each row and column line. Compared with the prior art, the present application omits the complex fitting process, improves the table structuring efficiency, eliminates the additional error caused by the fitting process, and improves the accuracy of the structured table generated by the curve function of the row and column line. Further, the subsequent extraction, analysis and processing of the table data are facilitated.

[0078] In addition, in the modeling method of the prior art, the connection between the key points can not be smooth, and jagged lines can easily occur. The continuity and smoothness of the row and column lines finally fitted depend on the post-processing fitting algorithm. In this embodiment, the row and column lines are directly defined as a curve function for describing a smooth curve, and the finally generated line is naturally continuous and smooth.

[0079] Next, through the following embodiments, other possible implementations of the table structuring method provided in the present application are described.

[0080] Reference Figure 4 The process of training the control parameter prediction model in the present embodiment includes the following steps:

[0081] Step S200, obtain a table image sample and a control parameter label corresponding to the table image sample.

[0082] The main purpose of this step is to collect a data set for training the model, which includes a table image sample and a control parameter label corresponding to the table image sample. The table image sample is an image sample containing a table, which is an object that the model needs to learn to process. The control parameter label is a set of control parameters corresponding to each row and column line in the table image sample after converting the row and column lines in the table image sample into a curve function, which is the expected output result of these table image samples and provides supervision information for model training, enabling the model to learn the mapping relationship from the table image to the set of control parameters of the row and column lines in the table image.

[0083] The table image sample can be collected in various ways, such as scanning an electronic table image from a paper document or directly extracting a table image from an electronic document. In addition to this, considering the diversity of images in the table image sample, the manual annotation method has high step cost and may not guarantee the consistency of the control parameters of the same table image sample. Therefore, in some possible implementations, the present application can annotate the table image sample according to the set annotation rules by the computer device, so that the annotation results of the table image sample are as consistent as possible.

[0084] Specifically, taking the process of annotating a set of control parameters corresponding to a row and column line in the table image sample as an example, the process of obtaining the table image sample and the control parameter label corresponding to the table image sample is described. Detect the table image sample, identify each row and column line, and refer to Figure 3 , where the solid line is a row line in the identified table, and one or more coordinate points P i (x i ,y i ) on the row line are taken. Substitute the coordinate points P i (x i ,y i ) into the predefined curve function, such as a cubic Bezier curve, to obtain the equation (2) containing a set of unknown control parameters P0, P1, P2, P3. To solve the four unknown control parameters, four coordinate points can be taken from the row line to construct an equation group to solve P0, P1, P2, P3. It can be understood that different orders of Bezier curve functions or different curve functions used will have different numbers of unknown control parameters, and the number of selected known coordinate points will also be different. In the actual annotation process, the corresponding number of coordinate points can be selected according to the number of unknown control parameters in the selected curve function, which will not be exemplified one by one.

[0085] (2)

[0086] Further, solving the equation containing a set of unknown control parameters P0, P1, P2, P3, the control parameters P0, P1, P2, P3 for describing the row and column line where the coordinate point P i (x i ,y i ) is located are obtained. According to the above manner, a set of control parameters corresponding to each row and column line is calculated, and all the row and column lines in the table image sample are marked with a set of control parameters corresponding to each row and column line.

[0087] In step S210, image features of the table image sample are extracted, and a control parameter prediction result is obtained based on the image features.

[0088] This step is the core processing part of the control parameter prediction model. First, meaningful image features are extracted from the table image sample. These image features can capture key contents such as the structure, layout, and text information of the table. Then, a prediction result of the control parameter is obtained based on the extracted image features.

[0089] Referring to Figure 5 , a schematic diagram of a control parameter prediction model training framework provided by the embodiments of the present application is provided. In the control parameter prediction model training process, at least the following modules are involved: a feature extraction module and a control parameter prediction module.

[0090] The feature extraction module is used to extract features from the table image sample to obtain image features. The extracted image features are used as the input of the control parameter prediction module.

[0091] The feature extraction module can use a convolutional neural network (CNN) in deep learning to extract features. For example, a pre-trained VGG, ResNet, or other model is used as a feature extractor. The table image sample is input into these models, and the output of the last layer or an intermediate layer is obtained as the image features. Alternatively, a special feature extraction network is designed according to the characteristics of the table image, such as feature extraction for lines, cells, and other structures of the table.

[0092] In one possible implementation, the embodiments use a convolutional neural network (such as ResNet50, Swin Transformer, etc.) that is pre-trained on the ImageNet dataset as a visual feature extractor to construct the feature extraction module. With the ability to learn general visual representations on a large number of natural images, the feature extraction module can efficiently map the input table image to a high-dimensional feature space, thereby capturing key image information such as basic textures, edge contours, and semantic structures in the image.

[0093] To further excavate the multi-level and multi-scale image features of the table image, the different expression capabilities of the different deep layers of the network are fully utilized, and the outputs of the first four layers are particularly selected as image features, which are denoted as F1, F2, F3, and F4. The F1 and F2 of the shallow layer mainly extract the bottom-level detail features of the table image, such as line texture and local edge, and the F3 and F4 of the deep layer are more inclined to capture the high-level features of the semantic layer, such as the overall layout of the table and the cell distribution pattern.

[0094] Further, considering the differences in resolution and channel number of the feature maps output by different layers, the bilinear interpolation, maximum pooling layer and other sampling technologies are used to adjust the F1, F2, F3 and F4 to the same resolution size, and the processed features are spliced in the channel dimension to realize effective fusion of the features and obtain the fusion feature F. The next step of control parameter prediction is performed according to the fusion feature F. It can be understood that this multi-scale feature fusion strategy can integrate multi-level information from the bottom layer to the high layer, retain the fine structure information of the table image, and fuse the global semantic layout features to improve the accuracy of feature expression and further optimize the accuracy of the prediction result.

[0095] In another possible implementation, the table image is input into the configured control parameter prediction model to obtain a group of control parameters corresponding to each row and column line in the table image, including: extracting image features of the table image; calculating attention information on the image features through the attention mechanism; and obtaining a group of control parameters corresponding to each row and column line in the table image based on the attention information.

[0096] Specifically, the feature extraction network described above can be used as the backbone network of the feature extraction module to perform multi-scale feature extraction on the table image and capture visual information including lines and overall layout in the table image. Meanwhile, the feature extraction module integrates the attention mechanism network, and the function of the attention mechanism network is to determine that the key area of the table row and column line should be focused on at the current feature extraction time, and to enhance the perception ability of the model to the table row and column line and the bending details of the table row and column line.

[0097] The image features extracted by the backbone network are further processed through the attention mechanism network, and the obtained attention information mainly includes the row and column line features in the image features. Further, a group of control parameters corresponding to the row and column line is predicted according to the row and column line features.

[0098] It can be understood that the training stage described in step S210 is also applicable to the inference stage, that is, the process of predicting the table image by the trained control parameter prediction model.

[0099] Step S220, calculate a first loss value based on the control parameter prediction result and the control parameter label.

[0100] Step S230, update the model parameters according to the total loss value until a set training end condition is reached, the total loss value at least including the first loss value L1.

[0101] Optionally, the mean square error between the control parameter prediction result and the control parameter label is calculated as the first loss value L1 to measure the difference between the prediction result of the control parameter prediction model and the true label, and the parameters of the model are updated using an optimization algorithm so that the prediction result of the model is closer to the true label. Through continuous iteration of this process, until the set training end condition is met, such as reaching the maximum number of iterations, the total loss value converging to a certain range, etc., a control parameter prediction model with better performance is obtained.

[0102] The training process of the above control parameter training model can fully utilize the information in the data by collecting and utilizing a large number of table image samples and corresponding labels, and can mine the potential rules between the table image and the control parameter, thereby obtaining more accurate prediction results. Further, by calculating the loss value and updating the model parameters, the model can continuously learn and improve, and gradually improve the understanding and prediction ability of the table image. With the training, the performance of the model will be continuously improved, and the model can better adapt to different table image and control parameter requirements.

[0103] In addition, the control parameter prediction model estimates the control parameters that control the shape and position of the curve function by recognizing the image features in the table image, and determines the specific curve function representing the row and column lines according to the control parameters output by the model. Compared with the method of extracting pixel key points to fit the row and column lines in the prior art, the present application estimates the control parameters by recognizing the overall features in the image, rather than relying only on local key points in the image, which greatly reduces the recognition error caused by image noise and other factors.

[0104] In one possible implementation, the training process of the control parameter prediction model can further include: predicting the key control parameters of each row and column line based on the image features, the key control parameters being the control parameters that determine the starting position of the curve represented by the curve function; calculating a second loss value L2 according to the predicted key control parameters of each row and column line and the key control parameters contained in the group of control parameters corresponding to each row and column line in the control parameter prediction result; and the total loss value further including the second loss value L2.

[0105] Referring to Figure 6 Another control parameter prediction model training framework provided by the embodiment of the present application is illustrated in the schematic diagram, and the training process of the control parameter prediction model is described in detail.

[0106] The image features of the table image sample extracted by the feature extraction module are taken as inputs of the key control parameter prediction module, and the key control parameter prediction module is used for predicting key control parameters of each row and column line of the table image sample, wherein the key control parameter is a control parameter for determining the starting position of a curve represented by a curve function. In the case of the curve function being a Bezier curve, the key control parameter is the first control point in a group of control points, which can also be referred to as a starting point P0.

[0107] It can be understood that the key control parameter (or starting point) is a control parameter for determining the starting position of a curve represented by a curve function, which to some extent restricts the starting direction and form of the curve. Therefore, the key control parameter prediction module is jointly trained with the control parameter prediction model (the feature extraction module and the control parameter prediction module) in the embodiment, and during the joint training process, the prediction result of the key control parameter prediction model is taken as a supervision signal for the prediction result of the control parameter, so as to provide more accurate positioning parameters and optimize the performance of the control parameter prediction model.

[0108] In one scenario, the row and column lines of the table in the table image are overlapped due to factors such as shooting angle and device resolution, so that the control parameter prediction model may be difficult to accurately identify and distinguish these overlapped row and column lines. To this end, the deviation between the number of control parameter groups in the control parameter prediction result and the total number of key control parameters of the key control parameter prediction model is taken as a second loss value, so that the control parameter prediction model can learn how to correctly identify the row and column lines in this complex situation. Even if the row and column lines are overlapped, the control parameter group corresponding to each row and column line can be accurately judged, the missed detection situation is reduced, and the adaptability of the model to complex image scenarios is greatly improved.

[0109] In another scenario, the control parameter prediction result contains multiple control parameter groups, and each control parameter group corresponds to a row and column line. The number of control parameter groups should be equal to the total number of row and column lines in the table, that is, equal to the total number of key control parameters. However, the row and column lines in the table image are affected by factors such as noise and interference lines, and there are situations where the row and column lines are blurred, which leads to the problem that even if the number of control parameter groups obtained by prediction is equal to the total number of row and column lines, the model still has a problem of ambiguity in positioning the starting point of the row and column line, that is, the problem of inaccurate prediction of the key control parameter.

[0110] To this end, by matching the key control parameter in each group of control parameters in the control parameter prediction result with the key control parameter predicted by the key control parameter prediction module, the key control parameter of each row and column line can be accurately located. Further, the distance deviation between the two key control parameters matched is calculated as a second loss value, so that the control parameter prediction model can more accurately measure the difference between the predicted key control parameter (the key control parameter in the control parameter prediction result) and the reference key control parameter (the key control parameter predicted by the key control parameter prediction module). During the training process, the second loss value is introduced to force the control parameter prediction model to pay more attention to the key control parameter of each row and column line, reduce the ambiguity of positioning, and make the model accurately identify the key control parameter of the row and column line under interference, greatly improving the positioning accuracy of the row and column line.

[0111] The training process of the control parameter prediction model provided by the embodiment can bring significant advantages to the control parameter prediction model used for inference by jointly training the control parameter prediction model (the feature extraction module and the control parameter prediction module) with the key control parameter prediction module during the training stage, i.e., using a multi-task training strategy (control parameter prediction task and key control parameter prediction task).

[0112] In summary, the control parameter prediction model training process provided by the above embodiments of the present application uses the key control parameter prediction task of the key control parameter prediction module for the table image sample as a strong supervision signal and guidance to improve the intrinsic ability of the control parameter prediction model. Finally, in the inference stage, even if the control parameter prediction model removes the key control parameter prediction module, the benefits brought by the training have been fixed in the remaining control parameter prediction model, so that the performance of the final lightweight control parameter prediction model is significantly better than that of the model trained only with the control parameter label.

[0113] In another possible implementation, the training process of the control parameter prediction model can further include: generating a first curve function of each row and column line based on the control parameter prediction result and the curve function, and determining a first table structure corresponding to the control parameter prediction result according to the first curve function; generating a second curve function of each row and column line based on the control parameter label and the curve function, and determining a second table structure corresponding to the control parameter label according to the second curve function; calculating a structure similarity loss value based on the similarity between the first table structure and the second table structure; and the total loss value further includes the structure similarity loss value.

[0114] The core of this training link is not only to supervise the model at the parameter level of the curve function, but also to supervise the model at the structure level.

[0115] Specifically, first, the control parameter prediction result predicted by the control parameter prediction model is rendered into a specific form of a row and column line through a predefined curve function (such as a Bezier curve), that is, a "first curve function", which is equivalent to converting abstract control parameter values into specific geometric figures. Further, the rendered row and column line ("first curve function") is combined to reconstruct a complete table structure, which can be denoted as "first table structure", which can clearly indicate the position, shape and adjacent relationship of all cells. Similarly, for multiple sets of control parameters in the control parameter label and the same curve function, the "second table structure" is rendered and reconstructed.

[0116] From the perspective of overall topology and geometric form, the similarity between the two reconstructed "first table structure" and "second table structure" is calculated as a structure similarity loss value. This structure similarity loss value, together with the first loss value and / or the second loss value, forms the total loss, which is used to update the model parameters of the control parameter prediction model.

[0117] The introduction of the structure similarity loss value enables the control parameter prediction model to move from local parameter optimization to global structure optimization, which can bring significant advantages to the control parameter prediction model used for inference, specifically in the following aspects:

[0118] 1. Improve the overall correctness of the table structure. The first loss value L1 and the second loss value L2 can only ensure that the predicted parameters of a single line are close to the label value, but even if the error of each line is small, the cumulative parameter error can lead to serious structural errors. For example, two lines that should be connected fail to connect due to slight deviation in the end point prediction, resulting in a single cell being incorrectly divided into two or two cells being incorrectly merged into one.

[0119] The structure similarity loss value directly targets whether the final output table structure is correct, and it can sensitively capture such topological errors and serious geometric deviations and feed them back to the model through the loss function, forcing the model to learn how to produce "structurally correct" predictions, rather than just "parameter values close to the level of individual line predictions". This greatly improves the usability and reliability of the final analysis results.

[0120] 2. Enhance the robustness and generalization ability of the model. Different table images may correspond to table forms with different parameter values but the same topology structure. For example, a table is translated, scaled or slightly rotated, and all the control parameter values of the table are changed, but the table structure remains unchanged. A model that simply uses parameter loss (L1, L2) may consider this as two completely different samples, thereby reducing the generalization ability. The structural similarity loss value pays more attention to the identity of the structure rather than the absolute value of the parameter. It guides the model to learn the concept of "structural equivalence", so that the model has natural robustness to geometric transformations and can better generalize to tables of different styles, sizes and directions.

[0121] In summary, by introducing the "structural similarity loss value", the supervision signal is elevated from low-level numerical regression to high-level semantic structure understanding, enabling the model to learn the high-level semantic features of the table grid formed by the row and column lines in the table image. The learning goal is directly aligned with the final task of table structuring, i.e., generating a correct table grid structure. Compared with existing technologies, when facing image noise and other disturbances, the model can judge the true shape of the table from a semantic level, reducing the impact of local parameter changes caused by noise, thereby significantly improving the overall accuracy, robustness and generalization of the parsing result.

[0122] Further, the control parameter prediction model trained through the above training process is applied to the inference stage of steps S100-S130 to obtain a set of control parameter prediction results for the table row and column lines. By accurately calculating the intersection of each pair of lines, the basic grid structure of the table is constructed, and the structured table is obtained.

[0123] In one possible implementation, for reconstructing the cells that are incorrectly split due to the intersection of row and column lines, after obtaining the structured table in step S130, the method further includes: generating a first adjacency graph of the table image and a second adjacency graph of the structured table based on the adjacency relationship between the smallest cells in the table image and the structured table, respectively; comparing the adjacency relationship difference between each corresponding smallest cell in the first adjacency graph and the second adjacency graph to generate a cell merging matrix, each value in the cell merging matrix representing the merging direction of the smallest cell at its corresponding position; and merging the smallest cells in the structured table according to the cell merging matrix to obtain the target structured cell.

[0124] Reference Figure 7An example diagram of cell merging is provided in the embodiments of the present application. In the process of structuring, each line intersection predicted by the model results in the cell being incorrectly split, such as the cross-cell Cell2 being incorrectly divided into two minimum cells. To this end, the embodiments merge the incorrectly split cells in the structured table based on the adjacency relationship between the cells in the structured table and the table image, and obtain the target structured cells consistent with the table image.

[0125] Specifically, first, a first adjacency graph of the table image is constructed, and a second adjacency graph of the structured table is constructed. Referring to the first adjacency graph in Figure 7 The adjacency graph is a graphical tool for describing the topological relationship between the minimum cells in the table. Through the connection relationship of "nodes" and "edges", the "nodes" represent the minimum cells, and the connection line between the two nodes, i.e. the "edge", represents the adjacent relationship between the two minimum cells corresponding to the two nodes.

[0126] Subsequently, the actual adjacency relationship between each cell and its adjacent cells in the structured table is calculated by comparing the first adjacency graph and the second adjacency graph. For example, Cell1-1 and Cell1-2 correspond to Cell1, which should logically belong to the same cell and jointly adjoin Cell2-1 and Cell3; Cell2-1 and Cell2-2 correspond to Cell2, which should logically belong to the same cell and jointly adjoin Cell3 and Cell4; Cell3 and Cell4 are independent. Based on this, a cell merging matrix of the structured table is generated, obtaining a 2x3 matrix as shown in equation (3). Each value in the cell merging matrix represents the merging direction of the minimum cell at the corresponding position, such as "1" representing merging to the left, "2" representing merging upwards, and "0" representing no merging.

[0127] (3)

[0128] Further, the cell merging matrix is traversed to identify whether the values corresponding to the cells to the right and below the current cell satisfy the merging condition, i.e. whether they are "1" or "2". If so, the current cell is merged to the right or downwards. Alternatively, only the value corresponding to the current cell is detected to determine whether it satisfies the merging condition, i.e. whether the value corresponding to the current cell is "1" or "2". If so, the current cell is merged to the left or upwards, otherwise the cell remains independent. Based on this, each minimum cell in the structured table is merged to obtain the target structured table with the same cell adjacency relationship as the table image, thereby restoring the logical structural integrity of the table.

[0129] Referring to Figure 8The application embodiment provides an application example diagram of a table structuring method, takes a Bezier curve as a preset curve function, and illustrates a training of a control parameter prediction model and a table structuring process.

[0130] The training of the control parameter prediction model is illustrated. The control parameter prediction model comprises a feature extraction network, a row line prediction module, and a column line prediction module. The feature extraction network is used to extract image features of an input table image. The row line prediction module is used to predict a group of control points corresponding to each row line based on the image features. The column line prediction module is used to predict a group of control points corresponding to each column line based on the image features.

[0131] In the training stage, the row line prediction module is jointly trained with the row line start point prediction module, and the column line prediction module is jointly trained with the column line start point prediction module. In the joint training process, the predicted row line start points and column line start points are used to supervise the output results of the row line prediction module and the column line prediction module, respectively.

[0132] It can be understood that the feature extraction network corresponds to the feature extraction module in the foregoing description, the row line prediction module and the column line prediction module correspond to the control parameter prediction module in the foregoing description, the feature extraction network, the row line prediction module, and the column line prediction module constitute the control parameter prediction model, and the row line start point prediction module and the column line start point prediction module correspond to the key control parameter prediction module in the foregoing description. For the training of the control parameter prediction model in the example and the joint training process of the control parameter prediction model, the row line start point prediction module, and the column line start point prediction module, refer to the foregoing description, and details are not described herein.

[0133] The inference stage of the control parameter prediction model trained according to the foregoing description is illustrated. A table image to be structured is input into the control parameter prediction model, the image features of the table image are extracted by the feature extraction network, the image features are used as the row line prediction module and the column line prediction module for prediction processing, a group of control points corresponding to each row line and a group of control points corresponding to each column line in the table image are obtained. According to the group of control points corresponding to each row line and column line, the Bezier curve function formula is substituted into the Bezier curve corresponding to each row line and column line. According to the Bezier curve corresponding to each row line and column line, the table is drawn to obtain a structured table.

[0134] Further, a cell adjacency graph corresponding to the table image and the structured table is constructed respectively, then, according to the cell adjacency graph, a topological adjacency relationship between each cell and its adjacent cells is calculated, and a merging decision matrix corresponding to the structured table is generated. The merging decision matrix is traversed, for each cell, if the cell below satisfies the merging condition, such as the corresponding value of the cell below in the merging decision matrix is "2", the cell is merged downward. If the cell on the right satisfies the merging condition, such as the corresponding value of the cell on the right in the merging decision matrix is "1", the cell is merged to the right. If neither satisfies the merging condition, it indicates that the cell is an independent cell, and remains independent. Finally, according to the merging decision matrix, a merging operation is recursively performed, and the logical structure integrity of the table is recovered.

[0135] An electronic device is also provided in the embodiments of the present application. Referring to Figure 9 Fig. 1 shows a structural schematic diagram suitable for implementing the electronic device in the embodiments of the present application. The electronic device in the embodiments of the present application can include, but is not limited to, a terminal such as a mobile phone, a tablet computer, a learning machine, a teaching large screen, a robot, a wearable device, and the like. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0136] As shown in Figure 9 The electronic device can include a processing device (for example, a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 2 or loaded from a storage device 8 to a random access memory (RAM) 3, to implement the table structuring method of the aforementioned embodiments of the present application. In the state that the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 3. The processing device 1, the ROM 2, and the RAM 3 are connected to each other through a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0137] Generally, the following devices can be connected to the I / O interface 5: input devices 6 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 7 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 8 including, for example, a memory card, a hard disk, and the like; and communication devices 9. The communication devices 9 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 9 The electronic device with various devices is shown, but it should be understood that all the devices shown are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0138] The embodiment of the present application further provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any one of the table structuring methods provided by the embodiment of the present application.

[0139] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the table structuring methods provided by the embodiment of the present application.

[0140] In addition, it should be noted that the above-described apparatus embodiments are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the methods described in various embodiments of the present application.

[0142] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part.

[0143] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0144] The various embodiments in the specification are described in a progressive manner, each embodiment focuses on the difference from other embodiments, the various embodiments can be combined as needed, and the same or similar parts refer to each other.

Claims

1. A tabular structuring method characterized by, The method comprises: obtaining a table image to be structured and a preset curve function comprising a set of unknown control parameters, the control parameters being used to control the shape and position of the curve represented by the curve function; inputting the table image into a configured control parameter prediction model to obtain a set of control parameters corresponding to each row and column line in the table image, the control parameter prediction model being trained based on table image sample data labeled with a set of control parameters corresponding to each row and column line; generating a curve function corresponding to each row and column line based on the set of control parameters corresponding to each row and column line and the curve function; generating a structured table of the table image according to the curve function corresponding to each row and column line.

2. The tabular structured method of claim 1, wherein, The training process of the control parameter prediction model comprises: obtaining a table image sample and its corresponding control parameter label, the control parameter label comprising a set of control parameters corresponding to each row and column line in the table image sample; extracting image features of the table image sample and predicting control parameter prediction results based on the image features; calculating a first loss value based on the control parameter prediction results and the control parameter label; updating model parameters according to a total loss value until a set training end condition is reached, the total loss value comprising at least the first loss value.

3. The tabular structuring method according to claim 2, characterized in that, The training process of the control parameter prediction model further comprises: predicting key control parameters of each row and column line based on the image features, the key control parameters being control parameters that determine the starting position of the curve represented by the curve function; calculating a second loss value based on the predicted key control parameters of each row and column line and the key control parameters contained in the set of control parameters corresponding to each row and column line in the control parameter prediction results; the total loss value further comprises the second loss value.

4. The tabular structuring method according to claim 2, characterized in that, The training process of the control parameter prediction model further comprises: generating a first curve function of each row and column line based on the control parameter prediction results and the curve function, and determining a first table structure corresponding to the control parameter prediction results according to the first curve function; generating a second curve function of each row and column line based on the control parameter label and the curve function, and determining a second table structure corresponding to the control parameter label according to the second curve function; calculating a structure similarity loss value based on the similarity between the first table structure and the second table structure; the total loss value further comprises the structure similarity loss value.

5. The tabular structured method according to any one of claims 1 to 4, wherein, The curve function is a Bezier curve function, the set of control parameters is a set of control points in the Bezier curve function, and the key control parameter is the first control point in the set of control points.

6. The tabular structured method according to any one of claims 1 to 4, wherein, The method further comprises: generating a first adjacency graph of the table image and a second adjacency graph of the structured table based on the adjacency relationship between the smallest cells in the table image and the structured table, respectively; Differences between the adjacency relationships between each corresponding minimum cell in the first adjacency graph and the second adjacency graph are compared to generate a cell merging matrix, each value in the cell merging matrix representing a merging direction of a minimum cell at a corresponding position; According to the cell merging matrix, the minimum cells in the structured table are merged to obtain target structured cells.

7. The tabular structured method according to any one of claims 1 to 4, wherein, The control parameter prediction model is input with the table image to obtain a set of control parameters corresponding to each row and column line in the table image, including: extracting image features of the table image; attention information for the image features is calculated through an attention mechanism; based on the attention information, a set of control parameters corresponding to each row and column line in the table image is obtained.

8. An electronic device, comprising: comprising: a memory and a processor; the memory is configured to store a program; the processor is configured to execute the program to implement each step of the table structuring method according to any one of claims 1-7.

9. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement each step of the table structuring method according to any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement each step of the table structuring method according to any one of claims 1-7.

Images