Table restoration method, device, electronic device and storage medium

Abstract

The present invention provides a table restoration method, apparatus, electronic device, and storage medium. The method comprises: obtaining a target table image containing a target table; inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights; and obtaining table lines in the target table based on the feature vector with attention weights. The table restoration method provided by the present invention utilizes a model with an attention module for recognition and segmentation, thereby improving the recognition accuracy of non-table lines, enhancing table restoration effectiveness, and enhancing the user experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a table restoration method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of office software, people's requirements for office software processing are getting higher and higher. They hope to improve the quality of work content while meeting normal office requirements.

[0003] At present, although the processing technology for table images can meet people's work requirements, there are still certain shortcomings. For example, the existing technology based on fully convolutional network (FCN) technology in the restoration of image table lines can only restore all the contents in the table image, and cannot identify which are the lines in the table and which are not. As a result, it is difficult to identify non-table lines in the image during the restoration process, and it is easy to restore non-table lines into table lines, resulting in poor restoration effect and incomplete lines, which affects the user experience. Summary of the Invention

[0004] Based on the problems existing in the prior art, the present invention proposes a table line segmentation and table restoration method, device, electronic device and storage medium, which can meet the situation where non-table lines exist in the table image, realize accurate recognition of table lines, and have the advantages of improving the table image segmentation processing effect and enhancing user experience.

[0005] In a first aspect, the present invention provides a table restoration method, comprising:

[0006] Obtaining a target table image containing a target table;

[0007] The target table image is input into a pre-trained line segmentation model to obtain a feature vector with attention weights, and the table lines in the target table are obtained based on the feature vector with attention weights.

[0008] Furthermore, the method further comprises:

[0009] The line segmentation model includes an attention module, which is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector, and obtain a feature vector with attention weight based on the feature vector and the attention weight vector.

[0010] Furthermore, according to a table restoration method provided by the present invention, the attention module includes: a feature vector compression layer, an attention learning layer, a sampling layer, and a feature vector fusion layer; wherein,

[0011] The feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image;

[0012] The attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight to obtain an attention weight vector;

[0013] The sampling layer performs an upsampling operation on the attention weight vector to obtain an upsampled attention weight vector;

[0014] The feature vector fusion layer is used to fuse the first feature vector of the target table image with the upsampled attention weight vector to obtain a feature vector with attention weight.

[0015] Furthermore, according to a table restoration method provided by the present invention, the attention module further includes: a convolutional layer;

[0016] The convolution layer is used to perform a convolution operation on the third eigenvector of the target table image to obtain the first eigenvector of the target table image; wherein the third eigenvector of the target table image is obtained by performing feature extraction on the target table image.

[0017] Furthermore, according to a table restoration method provided by the present invention, inputting the target table image into a pre-trained line segmentation model to obtain table lines in the target table includes:

[0018] Inputting the first feature vector obtained after feature extraction of the target table image into the feature vector compression layer to obtain a compressed second feature vector of the target table image;

[0019] Inputting the second eigenvector of the target table image into the attention learning layer to obtain an attention weight vector;

[0020] Inputting the attention weight vector into the sampling layer to obtain an upsampled attention weight vector; wherein the upsampled attention weight vector has the same dimension as the first eigenvector of the target table image;

[0021] Inputting the upsampled attention weight vector and the first feature vector of the target table image into the feature vector fusion layer to obtain a feature vector with attention weight;

[0022] Based on the feature vector with attention weights, table lines in the target table are obtained.

[0023] Furthermore, according to the table restoration method provided by the present invention, obtaining the table lines in the target table based on the feature vector with attention weights includes:

[0024] Determining lines in the target table image based on the feature vector with attention weights;

[0025] Recognizing lines in the target table image;

[0026] In the case where the lines in the target table image are identified as table lines, confirming to segment the target table image to obtain table lines in the target table;

[0027] When it is recognized that the lines in the target table image are background lines, it is determined that the target table image is not to be segmented.

[0028] Furthermore, according to the table restoration method provided by the present invention, the method further includes:

[0029] reconstructing the target table according to the table lines of the target table;

[0030] Perform text recognition on the target table image, and fill the text recognition result into the reconstructed target table.

[0031] Furthermore, according to the table restoration method provided by the present invention, the line segmentation model includes a downsampling layer and an upsampling layer, wherein:

[0032] The downsampling layer includes a first convolution layer, a second convolution layer, a third convolution layer and a maximum pooling layer; the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer all contain the attention module;

[0033] The upsampling layer includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer and a fourth deconvolution layer; the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer all contain the attention module.

[0034] In a second aspect, the present invention further provides a table restoration device, comprising:

[0035] An acquisition module, used for acquiring a target table image containing a target table;

[0036] An input module is used to input the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtain table lines in the target table based on the feature vector with attention weights.

[0037] In a third aspect, the present invention further provides an electronic device comprising: a processor, a memory, and a bus, wherein:

[0038] The processor and the memory communicate with each other via the bus;

[0039] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the steps of any of the above table restoration methods.

[0040] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of any of the table restoration methods described above.

[0041] The present invention provides a table restoration method, apparatus, electronic device, and storage medium. These methods obtain a target table image containing a target table, input the target table image into a pre-trained line segmentation model, and generate a feature vector with attention weights. The table lines in the target table are then retrieved based on the feature vector with attention weights. The table restoration method provided by the present invention utilizes a model with an attention module for recognition and segmentation, improving the accuracy of non-table line recognition, enhancing table restoration performance, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flow chart of the table restoration method provided by the present invention;

[0044] Figure 2 This is an example diagram of a target table image provided by the present invention;

[0045] Figure 3 This is an example diagram of the processing result of the target table image in the prior art;

[0046] Figure 4 This is an example diagram of the image processing result with the attention module provided by the present invention;

[0047] Figure 5 Schematic diagram of the structure of the attention module provided by the present invention;

[0048] Figure 6It is a schematic diagram of the structure of the complete attention module provided by the present invention;

[0049] Figure 7 This is an overall flow chart of a table line segmentation method provided by the present invention;

[0050] Figure 8 It is a structural diagram of the line segmentation model provided by the present invention;

[0051] Figure 9 This is an example diagram of a target table image with a background provided by the present invention;

[0052] Figure 10 This is an example of the background line processing result with the attention module provided by the present invention;

[0053] Figure 11 This is a figure showing the processing result of a target table image with a background using the existing technology;

[0054] Figure 12 This is an example diagram of a target table image with cells having shadows provided by the present invention;

[0055] Figure 13 This is an example diagram of the shadow processing result with the attention module provided by the present invention;

[0056] Figure 14 This is an example diagram of the processing results of the three-line table with the attention module provided by the present invention;

[0057] Figure 15 It is a structural diagram of the table restoration device provided by the present invention;

[0058] Figure 16 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0060] Figure 1 A flow chart of a table restoration method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the table restoration method provided by the present invention includes the following steps:

[0061] Step 101: Acquire a target table image containing a target table.

[0062] In this embodiment, it is necessary to obtain a target table image containing a target table. The target table can be a table containing both table lines and non-table lines, or a table containing only table lines, or a table containing a background image, wherein non-table lines refer to lines that do not constitute cells, such as Figure 2 The horizontal line corresponding to the "Customer Signature" displayed in the upper left corner is a non-table line. It should be noted that the target table can be set according to the actual needs of the user and is not specifically limited here.

[0063] In this embodiment, the target table image refers to the table image processed by the method of the present invention. As the name suggests, a table image is a table in an image format, for example, a table saved in an image format such as jpg or png. It should be noted that the target table image can be a standalone table image or a table image generated by converting a table in a PDF file, and this is not specifically limited here.

[0064] Step 102: Input the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtain the table lines in the target table based on the feature vector with attention weights.

[0065] In this embodiment, the target table image with the target table obtained in step 101 needs to be input into a pre-trained line segmentation model. The line segmentation model processes the target table image to obtain a feature vector with attention weights, and then obtains the table lines in the target table based on the feature vector with attention weights. The target table may contain non-table lines, and the line segmentation model can identify the non-table lines and does not perform segmentation processing on them.

[0066] It should be noted that the line segmentation model is pre-trained based on the convolutional neural network (CNN) algorithm and the sample table image and table lines of the sample table. The line segmentation model is mainly composed of a convolution layer, a sampling layer and an activation layer.

[0067] For example, Figure 2 The target table image shown in FIG. 1 has non-table lines at both the upper left and upper right corners (e.g., Figure 2 As shown by the arrow in the middle), if the target table image is processed by the line segmentation model in the prior art, Figure 3 The processing result shown in FIG. 1 shows that the non-table lines in the upper left corner and the upper right corner are segmented; if the line segmentation model with the attention module provided by the present invention is used to segment the target table image, the obtained Figure 4As shown in the segmentation processing results, the line segmentation model with the attention module identifies the non-tabular lines and does not segment the non-tabular lines, thus ensuring the accuracy of the tabular line segmentation processing.

[0068] The present invention provides a table restoration method. This method obtains a target table image containing a target table, inputs the target table image into a pre-trained line segmentation model, obtains a feature vector with attention weights, and then, based on the feature vectors with attention weights, retrieves the table lines in the target table. The table restoration method provided by the present invention uses a model with an attention module for recognition and segmentation, improving the accuracy of non-table line recognition, enhancing table restoration effectiveness, and enhancing the user experience.

[0069] Based on any of the above embodiments, in this embodiment, the method also includes: the line segmentation model includes an attention module, the attention module is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector, and based on the feature vector and the attention weight vector, a feature vector with attention weight is obtained.

[0070] In this embodiment, the main purpose is to improve the line segmentation model. An attention module is added to the line segmentation model to identify non-table lines. The attention module is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector. Then, based on the feature vector and the attention weight vector, a feature vector with attention weight is obtained. According to the feature vector with attention weight, accurate segmentation of the table lines can be achieved.

[0071] According to the table restoration method provided by the present invention, by adding an attention module to the line segmentation model, it is possible to improve the accurate recognition and segmentation of table lines or non-table lines, thereby improving the accuracy of table restoration processing.

[0072] Based on any of the above embodiments, in this embodiment, the attention module includes: a feature vector compression layer, an attention learning layer, a sampling layer, and a feature vector fusion layer; wherein,

[0073] The feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image;

[0074] The attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight to obtain the attention weight vector;

[0075] The sampling layer performs an upsampling operation on the attention weight vector to obtain the upsampled attention weight vector;

[0076] The feature vector fusion layer is used to fuse the first feature vector of the target table image with the attention weight vector after the upsampling operation of the sampling layer to obtain a feature vector with attention weights.

[0077] In this embodiment, an attention module is added to the line segmentation model, and the attention module includes a feature vector compression layer, an attention learning layer, a sampling layer and a feature vector fusion layer, wherein the feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image, wherein the first feature vector refers to the feature vector that has not been compressed, and the second feature vector refers to the feature vector after compression, and the dimension of the first feature vector is higher than that of the second feature vector.

[0078] like Figure 5 As shown, if the first eigenvector obtained is U, U∈R H×W×C , where H, W and C represent the height, width and number of channels of the feature map of the target table image respectively, and F sq (·) indicates that the first eigenvector is compressed to obtain the compressed second eigenvector, that is, the first eigenvector of dimension H×W×C is compressed into the second eigenvector of dimension 1×1×C, where C i The eigenvector representing the i-th feature map has a larger value, indicating that the table line is more likely to be a table line. It should be noted that the compression process is to reduce the dimensionality of the first eigenvector, compressing the original high-dimensional first eigenvector into a second eigenvector in the format of 1×1×C.

[0079] In this embodiment, the attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight to obtain the attention weight vector. Figure 5 Medium F ex (; W) indicates that the low-dimensional second eigenvector is fully connected to obtain the attention weight vector. Full connection means that each node is connected to all nodes in the previous layer, integrating the features extracted previously.

[0080] In this embodiment, the sampling layer needs to perform an upsampling operation on the obtained low-dimensional attention weight vector to obtain an upsampled attention weight vector, which is a high-dimensional feature vector. Figure 5 Medium F scale (·,·) indicates that the sampling layer performs an upsampling operation on the obtained attention weight vector, that is, converting the 1×1×C attention weight vector into an H×W×C dimensional attention weight vector. The purpose is to keep the dimension of the attention weight consistent with the dimension of the feature vector. Upsampling refers to restoring the resolution of the feature map to the resolution of the original image.

[0081] In this embodiment, the feature vector fusion layer is used to fuse the first feature vector of the target table image with the attention weight vector after the sampling layer performs the upsampling operation to obtain a feature vector with attention weight. Figure 5 As shown in the figure, the first feature vector U is fused with the up-sampled attention weight vector C, and the fusion method is U i C i , after fusion processing, we get the feature vector with attention weight It should be noted that the attention weight represents the degree of attention to the features of the target table image. The larger the weight value, the greater the possibility that there are table lines in the target table. The smaller the weight value, the greater the possibility that the lines in the target table are non-table lines.

[0082] For example, if the first eigenvector U of the target table image is the following eigenvector, the first eigenvector U∈R 4×4×3 , specifically including the following contents:

[0083]

[0084]

[0085]

[0086]

[0087] Then, the feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image, wherein the second feature vector is expressed as: 10 8 6

[0091] Furthermore, the attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight, and obtain an attention weight vector, which is: 0.2 0.3 0.5

[0095] Using the sampling layer, the low-dimensional attention weight vector is upsampled to obtain the upsampled high-dimensional attention weight vector C. The high-dimensional attention weight vector C is as follows:

[0096] 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2

[0097] 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3

[0098] 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5

[0099] Finally, the feature vector fusion layer is used to fuse the first feature vector U of the target table image with the attention weight vector C after the upsampling operation of the sampling layer to obtain a feature vector with attention weights, that is, the feature vector with attention weights is:

[0100] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

[0101] 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.4

[0102] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

[0103] According to the table restoration method provided by the present invention, the attention module is set to include a feature vector compression layer, an attention learning layer, a sampling layer, and a feature vector fusion layer, and then each layer performs corresponding processing on the feature vector of the acquired target table image, and finally obtains a feature vector with attention weights, which can be used in the recognition and confirmation of non-table lines, thereby improving the accuracy of non-table line recognition and improving the efficiency of table line segmentation processing.

[0104] Based on any of the above embodiments, in this embodiment, the attention module further includes: a convolutional layer;

[0105] The convolution layer is used to perform a convolution operation on the third eigenvector of the target table image to obtain the first eigenvector of the target table image; wherein the third eigenvector of the target table image is obtained by performing feature extraction on the target table image.

[0106] In this embodiment, the attention module provided by the present invention further includes a convolution layer, which is used to perform a convolution operation on the third eigenvector of the target table image to obtain the first eigenvector of the target table image, wherein, Figure 6 As shown, the third eigenvector is X, and X∈R H×W×C The third eigenvector X is obtained by extracting features from the target table image. The convolution layer then performs a convolution operation on the third eigenvector X to obtain the first eigenvector U. The eigenvector compression layer compresses the first eigenvector U to obtain the second eigenvector of the target table image. The attention learning layer performs a full connection operation on the second eigenvector for learning the attention weight to obtain the attention weight vector. The sampling layer performs an upsampling operation on the attention weight vector to obtain the upsampled attention weight vector C. The eigenvector fusion layer is used to fuse the first eigenvector U with the upsampled attention weight vector C to obtain a eigenvector with attention weight.

[0107] According to the table restoration method provided by the present invention, a convolution operation is performed on the third eigenvector of the extracted target table image through a set convolution layer to obtain a first eigenvector, which is used in subsequent operation processing to identify the table lines in the target table and improve the accuracy of table line recognition in the target table.

[0108] Based on any of the above embodiments, in this embodiment, the target table image is input into a pre-trained line segmentation model to obtain a feature vector with attention weights, and table lines in the target table are obtained based on the feature vector with attention weights, including:

[0109] Inputting the first feature vector obtained after feature extraction of the target table image into the feature vector compression layer to obtain the second feature vector of the compressed target table image;

[0110] Input the second eigenvector of the target table image into the attention learning layer to obtain the attention weight vector;

[0111] Input the attention weight vector into the sampling layer to obtain an upsampled attention weight vector; wherein the upsampled attention weight vector has the same dimension as the first eigenvector of the target table image;

[0112] The upsampled attention weight vector and the first eigenvector of the target table image are input into the feature vector fusion layer to obtain a feature vector with attention weights;

[0113] Based on the feature vector with attention weights, the table lines in the target table are obtained.

[0114] In this embodiment, it is necessary to obtain the table lines in the target table based on the feature vectors with attention weights obtained after processing each layer in the attention module. It should be noted that there are two channels in the output of the line segmentation model, one channel is used to output the horizontal lines in the target table, and the other channel is used to output the vertical lines in the target table. The specific settings can be made according to the actual needs of the user, and no specific limitation is made to the channels here.

[0115] It should be noted that, in this embodiment, the first feature vector of the extracted target table image needs to be input into the feature vector compression layer in the attention module, and then the obtained second feature vector is input into the attention learning layer, and then the low-dimensional attention weight vector is input into the sampling layer in the attention module to obtain an attention weight vector with the same dimension as the first feature vector, and the upsampled attention weight vector and the first feature vector of the target table image are input into the feature vector fusion layer to finally obtain a feature vector with attention weights. In this embodiment, the table lines in the target table are obtained based on the feature vector with attention weights, and when the feature vector enters the preset channel, the horizontal or vertical lines corresponding to the channel are obtained. It should be noted that the above processing operations are implemented in the attention module.

[0116] According to the table restoration method provided by the present invention, the feature vectors extracted from the target table image are processed accordingly through the various layers in the attention module, and finally a feature vector with attention weight is obtained. The table lines in the target table are determined based on the feature vector with attention weight, thereby improving the accuracy of table line segmentation in the target table and enhancing the user experience.

[0117] Based on any of the above embodiments, in this embodiment, obtaining table lines in the target table based on the feature vector with attention weights includes:

[0118] Determine the lines in the target table image based on the feature vector with attention weights;

[0119] Recognize the lines in the target table image;

[0120] In the case where the lines in the target table image are identified as table lines, confirming to segment the target table image to obtain table lines in the target table;

[0121] When the lines in the target table image are identified as background lines, it is confirmed that the target table image is not to be segmented.

[0122] In this embodiment, before segmenting the target table image, it is necessary to judge whether the target table image needs to be segmented, and the lines in the target table image are identified and confirmed through the line segmentation model. When the lines in the target table image are identified as table lines, the target table image is determined to be segmented to obtain the table lines in the target table; when the lines in the target table image are identified as background lines, that is, there are no table lines in the target table image, the target table image is not segmented.

[0123] It should be noted that in this embodiment, the table lines and non-table lines are determined by the size of the attention weight. If the attention weight of a line is large and exceeds the preset threshold, the line is determined as a table line. If the attention weight of a line is small and is lower than the preset threshold, the line is determined as a non-table line.

[0124] It should be noted that, in other embodiments, table lines and non-table lines may be determined by other identification and confirmation methods, which may be set according to actual needs of the user and are not specifically limited here.

[0125] According to the table restoration method provided by the present invention, by identifying and confirming the lines in the target table image, it is possible to distinguish between table lines and non-table lines, thereby improving the accuracy of image segmentation and restoration.

[0126] Based on any of the above embodiments, in this embodiment, the table restoration method further includes:

[0127] Reconstruct the target table according to its table lines;

[0128] Perform text recognition on the target table image and fill the text recognition results into the reconstructed target table.

[0129] In this embodiment, the provided table restoration method is primarily used to restore a target table in a target table image and rebuild a new target table. Currently, the target table in the target table image is in an uneditable state. To implement data editing operations in the target table, the target table needs to be restored. Table restoration refers to restoring the target table in the target table image into an editable table. A new target table is constructed based on the table lines of the target table obtained above, and the text content in the target table image is filled into the reconstructed target table to obtain a new target table.

[0130] It should be noted that in this embodiment, when reconstructing the target table, it is also necessary to recognize the text in the target table image and fill the text recognition results into the reconstructed target table. Among them, in this embodiment, the text recognition of the target table image and the filling of the text recognition results into the reconstructed target table both adopt the more mature processing technology in the existing technology, which will not be introduced in detail here.

[0131] According to the table restoration method provided by the present invention, by reconstructing the target table based on the table lines of the segmented target table and filling the text recognition results of the target table image into the constructed new target table, the target table can be accurately restored and the processing speed of the target table can be improved.

[0132] Based on any of the above embodiments, in this embodiment, the line segmentation model includes a downsampling layer and an upsampling layer, wherein:

[0133] The downsampling layer includes the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer; the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer all contain attention modules;

[0134] The upsampling layer includes the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer; the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer all contain attention modules.

[0135] In this embodiment, the line segmentation model includes an upsampling layer and a downsampling layer. Figure 7 As shown, since the line segmentation model is a Unet model, which is mainly a convolution model, the downsampling layer includes a first convolution layer, a second convolution layer, a third convolution layer and a maximum pooling layer, and the upsampling layer includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer and a fourth deconvolution layer. In addition, in this embodiment, an attention module is provided in the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer, the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer for distinguishing between table lines and non-table lines in the target table image.

[0136] According to the table restoration method provided by the present invention, by providing an attention module in each layer in the upsampling layer and the downsampling layer, the type of lines in the target table image can be accurately identified, thereby improving the accuracy of table restoration.

[0137] Based on any of the above embodiments, in this embodiment, a target table image containing a target table is first obtained, and the target table image is input into a pre-trained line segmentation model, which is a line segmentation Unet model with attention, and then the horizontal and vertical lines of the target table are output.

[0138] In this embodiment, an improvement is made to the line segmentation model built on a convolutional neural network algorithm by adding an attention module within the UNet model. This module extracts features from the target table image. The more complete the feature extraction, the more effective the segmentation of table lines. This allows the line segmentation model to automatically identify non-table lines and not segment them.

[0139] In this embodiment, the attention module includes a convolution layer, a feature vector compression layer, an attention learning layer, a sampling layer, and a feature fusion layer. The third feature vector of the target table image is extracted as X, and X∈R H×W×C , perform convolution operation on X to obtain the first eigenvector U, U∈R H×W×C,W,and C,are the height, width and number of channels of the target table image, respectively, and,F tr Represents the convolution mapping of X→U, and then the feature vector U obtained by convolution needs to be compressed, F sq (·) represents the compression processing of the eigenvector, and the second eigenvector with 1×1×C dimensions is obtained. That is, the first eigenvector with H×W×C dimensions is compressed into the second eigenvector with 1×1×C dimensions, where C i Represents the attention weight of the i-th feature map. And perform full connection processing on the second feature vector to obtain the attention weight vector, and then pass F scale (·,·) performs upsampling on the attention weight vector to obtain an attention weight vector C with the same dimension as the first eigenvector, and then fuses the first eigenvector U with the attention weight vector C to obtain a eigenvector with attention weight.

[0140] It should be noted that X represents the third eigenvector of the extracted target table image, and the first eigenvector U is obtained by performing a convolution operation on the third eigenvector. The first eigenvector U can also be obtained without performing a convolution operation on the third eigenvector, or can be obtained by directly performing feature extraction based on the target table image. The specific setting can be based on the actual needs of the user and is not specifically limited here.

[0141] In this embodiment, if Figure 7 As shown in the figure, the attention module exists in the position indicated by the arrow in the line segmentation model. It should be noted that the attention module is used in the entire line segmentation model, and each convolutional neural network operation contains an independent attention module.

[0142] In this embodiment, a line segmentation model with attention weights can also be used to reduce the influence of the table background on the table line segmentation. Figure 8 The target table image shown in FIG. 1 contains an image in each cell of the target table image. If the line segmentation model in the prior art is used for processing, the following is obtained: Figure 9 The processing result shown is obtained by processing the target table image with the line segmentation model with attention weight provided by the present invention. Figure 10 The processing result is shown in the figure. Under the influence of the background image, the existing technology does not accurately identify the non-table lines, resulting in Figure 9 The processing results in the above example segment the non-table lines, but the segmentation effect is not obvious and accurate. Figure 10 As shown, the method of the present invention can overcome the influence of the background image in the table on the table line segmentation.

[0143] In this embodiment, it can also be used to eliminate the background shadow in the cell, and obtain the following Figure 11The target table image shown is processed by the line segmentation model with attention weight provided by the present invention to obtain Figure 12 The processing results shown are clearly divided by lines in the table.

[0144] In this embodiment, it is also possible to process a target table image in the form of a three-line table, such as Figure 13 The target table image shown is processed by the line segmentation model with attention weight provided by the present invention to obtain Figure 14 The processing results shown are clearly divided by lines in the table.

[0145] Figure 15 The present invention provides a table restoration device, such as Figure 15 As shown, the table restoration device provided by the present invention includes:

[0146] An acquisition module 1501 is used to acquire a target table image including a target table;

[0147] The input module 1502 is used to input the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtain the table lines in the target table based on the feature vector with attention weights.

[0148] The present invention provides a table restoration device. This device obtains a target table image containing a target table, inputs the target table image into a pre-trained line segmentation model, and generates a feature vector with attention weights. Based on the feature vectors with attention weights, the table lines in the target table are then retrieved. The table restoration method provided by the present invention utilizes a model with an attention module for recognition and segmentation, improving the accuracy of non-table line recognition, enhancing table segmentation, and enhancing the user experience.

[0149] Furthermore, the table restoration device also includes: the line segmentation model includes an attention module, the attention module is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector, and based on the feature vector and the attention weight vector, a feature vector with attention weight is obtained.

[0150] According to the table restoration device provided by the present invention, by adding an attention module to the line segmentation model, the table lines and non-table lines of the target table image can be accurately identified, thereby improving the accuracy of the table restoration process.

[0151] Furthermore, the attention module includes: feature vector compression layer, attention learning layer, sampling layer, feature vector fusion layer; wherein,

[0152] The feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image;

[0153] The attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight to obtain the attention weight vector;

[0154] The sampling layer performs an upsampling operation on the attention weight vector to obtain the upsampled attention weight vector;

[0155] The feature vector fusion layer is used to fuse the first feature vector of the target table image with the attention weight vector after the upsampling operation of the sampling layer to obtain a feature vector with attention weights.

[0156] According to the table restoration device provided by the present invention, an attention module is set to include a feature vector compression layer, an attention learning layer, a sampling layer, and a feature vector fusion layer. Each layer then performs corresponding processing on the feature vector of the acquired target table image, and finally obtains a feature vector with an attention weight. This feature vector can be used in the recognition and confirmation of non-table lines, thereby improving the accuracy of non-table line recognition and the efficiency of table restoration processing.

[0157] Furthermore, the attention module also includes: convolutional layers;

[0158] The convolution layer is used to perform a convolution operation on the third eigenvector of the target table image to obtain the first eigenvector of the target table image; wherein the third eigenvector of the target table image is obtained by performing feature extraction on the target table image.

[0159] According to the table restoration device provided by the present invention, a convolution operation is performed on the third eigenvector of the extracted target table image through a set convolution layer to obtain a first eigenvector, which is used in subsequent operation processing to identify the table lines in the target table and improve the accuracy of table line recognition in the target table.

[0160] Furthermore, the input module 1502 is further configured to:

[0161] Inputting the first feature vector obtained after feature extraction of the target table image into the feature vector compression layer to obtain the second feature vector of the compressed target table image;

[0162] Input the second eigenvector of the target table image into the attention learning layer to obtain the attention weight vector;

[0163] Input the attention weight vector into the sampling layer to obtain an upsampled attention weight vector; wherein the upsampled attention weight vector has the same dimension as the first eigenvector of the target table image;

[0164] The upsampled attention weight vector and the first eigenvector of the target table image are input into the feature vector fusion layer to obtain a feature vector with attention weights;

[0165] Based on the feature vector with attention weights, the table lines in the target table are obtained.

[0166] According to the table restoration device provided by the present invention, the feature vectors extracted from the target table image are processed accordingly through the various layers in the attention module, and finally a feature vector with attention weight is obtained. The table lines in the target table are determined based on the feature vector with attention weight, thereby improving the accuracy of table line segmentation in the target table and enhancing the user experience.

[0167] Furthermore, the input module 1502 is further configured to:

[0168] Determine the lines in the target table image based on the feature vector with attention weights;

[0169] Recognize the lines in the target table image;

[0170] In the case where the lines in the target table image are identified as table lines, confirming to segment the target table image to obtain table lines in the target table;

[0171] When the lines in the target table image are identified as background lines, it is confirmed that the target table image is not to be segmented.

[0172] According to the table restoration method provided by the present invention, by identifying and confirming the lines in the target table image, the type of lines in the target table image can be accurately identified, thereby improving the accuracy of table restoration and enhancing user experience.

[0173] Furthermore, the table restoration device is also used to:

[0174] Reconstruct the target table according to its table lines;

[0175] Perform text recognition on the target table image and fill the text recognition results into the reconstructed target table.

[0176] According to the table restoration device provided by the present invention, by reconstructing the target table according to the table lines of the segmented target table based on the above-mentioned table line segmentation method, and filling the text recognition results of the target table image into the constructed new target table, it is possible to restore the target table and improve the processing speed of the target table.

[0177] Furthermore, the line segmentation model includes a downsampling layer and an upsampling layer, where

[0178] The downsampling layer includes the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer; the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer all contain attention modules;

[0179] The upsampling layer includes the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer; the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer all contain attention modules.

[0180] According to the table restoration device provided by the present invention, the line segmentation model is divided into an upsampling layer and a downsampling layer. By providing an attention module in each corresponding layer in the upsampling layer and the downsampling layer, the type of lines in the target table image can be accurately identified, thereby improving the accuracy of table restoration.

[0181] Since the principles of the apparatus of the embodiment of the present invention are the same as those of the method of the above embodiment, more detailed explanations are omitted here.

[0182] Figure 16 Schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 16 As shown, the present invention provides an electronic device, including: a processor (processor) 1601, a memory (memory) 1602 and a bus 1603;

[0183] The processor 1601 and the memory 1602 communicate with each other via the bus 1603.

[0184] Processor 1601 is used to call program instructions in memory 1602 to execute the methods provided in the above-mentioned method embodiments, for example, including: obtaining a target table image containing a target table; inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtaining table lines in the target table based on the feature vector with attention weights.

[0185] A computer-readable storage medium is provided in an embodiment of the present invention. The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to execute the methods provided in the above-mentioned method embodiments, for example, including: obtaining a target table image containing a target table; inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtaining table lines in the target table based on the feature vector with attention weights.

[0186] The present invention also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above methods, which include: obtaining a target table image containing a target table; inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtaining table lines in the target table based on the feature vector with attention weights.

[0187] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A table restoration method, characterized in that: include: Obtaining a target table image containing a target table; Inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtaining table lines in the target table based on the feature vector with attention weights; The line segmentation model includes an attention module, which is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector, and obtain a feature vector with attention weight based on the feature vector and the attention weight vector; when the attention weight of the line in the target table image exceeds a preset threshold, the line is a table line in the target table; when the attention weight of the line in the target table image is lower than a preset threshold, the line is not a table line in the target table.

2. The table restoration method according to claim 1, characterized in that: The attention module includes: a feature vector compression layer, an attention learning layer, a sampling layer, and a feature vector fusion layer; wherein, The feature vector compression layer is used to compress the first feature vector of the target table image to obtain the second feature vector of the target table image; The attention learning layer is used to perform a fully connected operation on the second eigenvector of the target table image for learning the attention weight to obtain an attention weight vector; The sampling layer is used to perform an upsampling operation on the attention weight vector to obtain an upsampled attention weight vector; The feature vector fusion layer is used to fuse the first feature vector of the target table image with the attention weight vector after the upsampling operation of the sampling layer to obtain a feature vector with attention weight.

3. The table restoration method according to claim 2, characterized in that: The attention module also includes: a convolutional layer; The convolution layer is used to perform a convolution operation on the third eigenvector of the target table image to obtain the first eigenvector of the target table image; wherein the third eigenvector of the target table image is obtained by performing feature extraction on the target table image.

4. The table restoration method according to claim 2, characterized in that: Inputting the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtaining table lines in the target table based on the feature vector with attention weights, includes: Inputting the first feature vector obtained after feature extraction of the target table image into the feature vector compression layer to obtain a compressed second feature vector of the target table image; Inputting the second eigenvector of the target table image into the attention learning layer to obtain an attention weight vector; Inputting the attention weight vector into the sampling layer to obtain an upsampled attention weight vector; wherein the upsampled attention weight vector has the same dimension as the first eigenvector of the target table image; Inputting the upsampled attention weight vector and the first feature vector of the target table image into the feature vector fusion layer to obtain a feature vector with attention weight; Based on the feature vector with attention weights, table lines in the target table are obtained.

5. The table restoration method according to claim 1, characterized in that: The obtaining of table lines in the target table based on the feature vector with attention weights includes: Determining lines in the target table image based on the feature vector with attention weights; Recognizing lines in the target table image; In the case where the lines in the target table image are identified as table lines, confirming to segment the target table image to obtain table lines in the target table; When it is recognized that the lines in the target table image are background lines, it is determined that the target table image is not to be segmented.

6. The table restoration method according to claim 1, characterized in that: The method further comprises: reconstructing the target table according to the table lines of the target table; Perform text recognition on the target table image, and fill the text recognition result into the reconstructed target table.

7. The table restoration method according to claim 1, characterized in that: The line segmentation model includes a downsampling layer and an upsampling layer, wherein: The downsampling layer includes a first convolution layer, a second convolution layer, a third convolution layer and a maximum pooling layer; the first convolution layer, the second convolution layer, the third convolution layer and the maximum pooling layer all contain the attention module; The upsampling layer includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer and a fourth deconvolution layer; the first deconvolution layer, the second deconvolution layer, the third deconvolution layer and the fourth deconvolution layer all contain the attention module.

8. A table restoration device, characterized in that: include: An acquisition module, used for acquiring a target table image containing a target table; An input module, configured to input the target table image into a pre-trained line segmentation model to obtain a feature vector with attention weights, and obtain table lines in the target table based on the feature vector with attention weights; The line segmentation model includes an attention module, which is used to perform attention learning on the feature vector of the target table image to obtain an attention weight vector, and obtain a feature vector with attention weight based on the feature vector and the attention weight vector; when the attention weight of the line in the target table image exceeds a preset threshold, the line is a table line in the target table; when the attention weight of the line in the target table image is lower than a preset threshold, the line is not a table line in the target table.

9. An electronic device, characterized in that: include: processor, memory, and bus, wherein The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the steps of the table restoration method according to any one of claims 1 to 7 by calling the program instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to execute the steps of the table restoration method according to any one of claims 1 to 7.

Images