A model training and table recognition method and device

Abstract

This specification discloses a model training and table recognition method and device. Several images containing tables are identified as training samples. For each training sample, the training sample's label is determined based on the structure and location of the table in the training sample. The training sample is input into the feature extraction layer of the recognition model. The image feature pyramid corresponding to the training sample is determined. For each feature map in the image feature pyramid, the reconstructed code corresponding to the feature map is determined. The reconstructed code corresponding to the feature map is upsampled and fused with other feature maps larger than the feature map. The fused results corresponding to each feature map are used as input and input into the recognition layer of the recognition model to obtain the recognition result for the training sample. This method fuses feature maps of different sizes to determine the recognition result of the training sample. The obtained image features are more comprehensive, and a rich amount of information can be obtained when recognizing the captured image, with high efficiency.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a model training and table recognition method and device. Background Art

[0002] Tables are widely used as an effective method for organizing and presenting data, and have become a common page object in various documents. With the explosive growth in the number of documents, efficiently finding tables in documents and extracting their content and structure—that is, table recognition—has become a pressing problem.

[0003] In the prior art, a common table recognition method is based on binarization. Specifically, an image containing a table can be first obtained and binarized. Then, the binarized image can be input into a recurrent neural network for row segmentation and a recurrent neural network for column segmentation, respectively, to obtain the row segmentation result and column segmentation result corresponding to the image. Finally, based on the row segmentation result and the column segmentation result, the semantic segmentation result of each pixel in the image containing the table can be determined, and the table can be determined based on the semantic segmentation result, wherein the semantic segmentation result includes whether it is the edge of the table or the edge of the cell.

[0004] However, the existing technology can recognize tables in scanned images, but when recognizing images captured by image sensors, the recognition effect is poor because the tables in the captured images usually contain distortion, occlusion, etc. Summary of the Invention

[0005] This specification provides a model training and table recognition method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a method for training a recognition model, including:

[0008] Determine a number of images containing tables as training samples, and for each training sample, determine a label for the training sample based on the structure and position of the table in the training sample;

[0009] The training sample is input into the feature extraction layer of the recognition model to be trained, and the training sample is upsampled and downsampled respectively to determine the image feature pyramid corresponding to the training sample;

[0010] For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, upsampling the reconstructed code corresponding to the feature map is performed, and the reconstructed code is fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result;

[0011] The fusion results corresponding to each feature map are input into the recognition layer of the recognition model to obtain the recognition result of the training sample output by the recognition layer;

[0012] The recognition model is trained based on the recognition results of each training sample and its annotations.

[0013] Optionally, the recognition result includes multiple types;

[0014] The recognition model is trained based on the recognition results and annotations of each training sample, specifically including:

[0015] For each training sample, determine the label corresponding to each type of the training sample according to the preset recognition result type and the label of the training sample;

[0016] According to the recognition results and labels of the various types corresponding to the training samples, a loss is determined, and model parameters of the recognition model are adjusted according to the loss.

[0017] Optionally, the fusion results corresponding to the feature maps are input as input to the recognition layer of the recognition model to obtain the recognition result of the training sample output by the recognition layer, specifically including:

[0018] The fusion results corresponding to the feature maps are input into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to the feature maps;

[0019] The fusion results of the same size corresponding to each feature map are input into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample.

[0020] Optionally, for each feature map in the image feature pyramid, the reconstructed code corresponding to the feature map is upsampled and fused with a feature map one level larger in size than the feature map to determine a fusion result.

[0021] Optionally, the recognition result includes twenty-nine types, and the combined layer is a twenty-nine-channel convolutional neural network layer, wherein five layers are used for semantic segmentation and twenty-four layers are used for pixel classification;

[0022] Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample specifically includes:

[0023] Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to determine the fusion features corresponding to the training sample;

[0024] According to the preset recognition result type and the fusion feature, the recognition result of the training sample output by each preset channel is obtained.

[0025] This specification provides a table recognition method, including:

[0026] Identify the image containing the table;

[0027] Taking the image as input, inputting it into the feature extraction layer of a pre-trained recognition model, upsampling and downsampling the image respectively, and determining an image feature pyramid corresponding to the image;

[0028] For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, upsampling the reconstructed code corresponding to the feature map is performed, and the reconstructed code is fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result;

[0029] The fusion results corresponding to each feature map are used as input and input into the recognition layer of the recognition model to obtain the recognition result of the image output by the recognition layer. According to the recognition result of the image, the structure and position of the table in the image are determined.

[0030] This specification provides a training device for a recognition model, including:

[0031] a sample determination module, configured to determine a number of images containing tables as training samples, and for each training sample, determine a label for the training sample based on the structure and position of the table in the training sample;

[0032] A sampling module is used to input the training sample as input to the feature extraction layer of the recognition model to be trained, upsample and downsample the training sample respectively, and determine the image feature pyramid corresponding to the training sample;

[0033] a fusion module, configured to perform convolution in different directions on each feature map in the image feature pyramid, determine a reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result;

[0034] The recognition module is used to input the fusion results corresponding to each feature map into the recognition layer of the recognition model, and obtain the recognition result of the training sample output by the recognition layer;

[0035] The training module is used to train the recognition model according to the recognition results and annotations of each training sample.

[0036] This specification provides a table recognition device, comprising:

[0037] A first determining module, configured to determine an image containing a table;

[0038] A second determination module is configured to take the image as input and input it into a feature extraction layer of a pre-trained recognition model, perform upsampling and downsampling on the image, and determine an image feature pyramid corresponding to the image;

[0039] a fusion module, configured to perform convolution in different directions on each feature map in the image feature pyramid, determine a reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result;

[0040] The third determination module is used to input the fusion results corresponding to each feature map into the recognition layer of the recognition model, obtain the recognition result of the image output by the recognition layer, and determine the structure and position of the table in the image based on the recognition result of the image.

[0041] This specification provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, any one of the above-mentioned recognition model training method or table recognition method is implemented.

[0042] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, any one of the above-mentioned recognition model training methods or table recognition methods is implemented.

[0043] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0044] In the training method of the recognition model provided in this specification, several images containing tables are determined as training samples. For each training sample, the label of the training sample is determined based on the structure and position of the table in the training sample. The training sample is input into the feature extraction layer of the recognition model, and the image feature pyramid corresponding to the training sample is determined. For each feature map in the image feature pyramid, the reconstructed code corresponding to the feature map is determined, and the reconstructed code corresponding to the feature map is upsampled and fused with other feature maps whose size is larger than the feature map. The fusion results corresponding to each feature map are used as input and input into the recognition layer of the recognition model to obtain the recognition result of the training sample.

[0045] It can be seen from the above method that this method is based on the fusion of feature maps of different sizes to determine the recognition results of training samples. The acquired image features are more comprehensive, and rich information is obtained at the moment of recognizing the collected images, which is highly efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0047] Figure 1 A flowchart of the training method for the recognition model provided in this specification;

[0048] Figure 2 A schematic diagram of the feature extraction layer of the recognition model provided in this specification;

[0049] Figure 3 A schematic diagram of determining the reconstruction code corresponding to each feature map provided in this specification;

[0050] Figure 4 A schematic diagram for determining fusion results provided in this specification;

[0051] Figure 5 A schematic diagram of the structure of the recognition model provided in this manual;

[0052] Figure 6 A flowchart of the table recognition method provided in this manual;

[0053] Figure 7 A schematic diagram of a training device for a recognition model provided in this specification;

[0054] Figure 8 A schematic diagram of the table recognition device provided in this manual;

[0055] Figure 9 The corresponding Figure 1or Figure 6 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0057] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0058] In the prior art, table recognition typically involves two application scenarios: recognizing PDF versions of tables and recognizing photos containing tables captured by a capture device. The amount of information about the table that can be determined from the image data corresponding to the scanned version and the image data corresponding to the image containing the table captured by the capture device is inconsistent. The scanned version of the image data contains much more information about the table than the image captured by the capture device.

[0059] Specifically, in the scenario of recognizing a scanned version of an image, the existing technology can usually recognize a relatively complete table. However, when recognizing an image captured by an acquisition device, the table in the captured image usually has a cluttered background, is contaminated, or is blocked, resulting in too little information about the table determined from the photo in the captured image, which in turn results in poor results when the existing technology recognizes the captured image.

[0060] Figure 1 This is a flow chart of the training method for the recognition model provided in this specification, which specifically includes the following steps:

[0061] S100: Determine a number of images containing tables as training samples, and for each training sample, determine a label for the training sample based on the structure and position of the table in the training sample.

[0062] Generally, in the field of table recognition, an image containing a table can be recognized through a recognition model to determine the position and structure of the table in the image, and then other steps can be performed based on the determined table.

[0063] Typically, a training model for a recognition model is pre-trained by a server for training the model based on training samples. This specification provides a method for training a recognition model, and similarly, the process of training the recognition model can be performed by a server for training the model.

[0064] The training model can be divided into a sample generation phase and a model training phase. In the sample generation phase, samples for training the model can be determined based on model requirements and training needs. In this specification, the server can first determine training samples for training the recognition model. Since the training model of the recognition model is generally based on images containing tables to recognize tables, the server can first determine each image containing a table to determine the training samples.

[0065] Based on this, the server may first obtain several images containing the form, wherein the image may include an image determined by scanning, or an image captured by an image capture device.

[0066] In the field of model training, model training is divided into two processes: determining samples and training models. In the process of determining samples, in addition to determining the samples used to train the model, it is also necessary to determine the labels of the training samples.

[0067] Specifically, the server may determine, for each training sample, a label for the training sample based on the structure and location of the table in the training sample. The server may directly use the structure and location of the table in each training sample as the label corresponding to each training sample, or may determine the label for each training sample based on the structure and location of the table in each training sample and a preset recognition structure type.

[0068] Of course, if the recognition result is the element to which each pixel in the training sample corresponds (a table border, a cell, outside the table, etc.), the annotation of the training sample can be the element to which each pixel corresponds, determined based on the table frame and position. If the recognition result is whether each pixel in the training sample belongs to the same type as the surrounding pixels, the annotation of the training sample can be whether each pixel in the training sample belongs to the same type as the surrounding pixels, determined based on the table frame and position. The form and type of the annotation of the training sample can be set as needed and this manual does not impose any restrictions on this.

[0069] S102: The training sample is input into a feature extraction layer of a recognition model to be trained, and the training sample is upsampled and downsampled respectively to determine an image feature pyramid corresponding to the training sample.

[0070] Unlike existing techniques that directly segment features of a binarized image to determine the segmentation results corresponding to the image features, which results in insufficient image information and poor recognition results, this specification provides a new method for training a recognition model. By determining an image feature pyramid for a training sample, determining a reconstructed code for the training sample, and then segmenting the image based on the reconstructed code, this method can obtain features with richer information from the image.

[0071] Based on this, the server may first take each training sample as input and input it into the feature extraction layer of the recognition model to be trained.

[0072] Then, the feature extraction layer of the recognition model can perform upsampling and downsampling of the training sample at different sizes to determine the image feature pyramid corresponding to the training sample. Figure 2 shown.

[0073] Figure 2 This is a schematic diagram of the feature extraction layer of the recognition model provided in this specification. The server can upsample and downsample the training samples input into the feature extraction layer, determine the upsampling results and downsampling results corresponding to the training samples, and determine the image feature pyramid corresponding to the training samples. The above-mentioned image feature pyramid is a sequence of pyramid feature maps composed of feature maps of different sizes. The size and number of upsampling and downsampling performed by the server, the methods used for upsampling and downsampling, and the number of layers of the image feature pyramid can be set as needed, and this specification does not impose any restrictions on this.

[0074] It should be noted that the reason for upsampling and downsampling the original image described above is that smaller images yield richer learned global-level texture features, while larger images yield richer learned pixel-level texture features. Therefore, based on learned global-level and pixel-level image features, a richer amount of information can be learned, enabling better recognition results based on the captured images.

[0075] S104: For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, and the reconstructed code corresponding to the feature map is upsampled and fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result.

[0076] In one or more embodiments provided in this specification, after determining the image feature pyramid corresponding to the training sample, the server may determine the reconstructed code corresponding to the training sample based on the image feature pyramid corresponding to the training sample.

[0077] Specifically, the server can perform convolution in different directions on each feature map in the image feature pyramid. The convolution in different directions includes at least two convolutions in opposite directions. For example, convolution is performed from top to bottom, then from bottom to top, then from left to right, and finally from right to left. When performing up and down convolution, the convolution kernel is a column vector, and when performing left and right convolution, the convolution kernel is a row vector, etc. Figure 3 shown.

[0078] Figure 3 A schematic diagram of determining the reconstruction code corresponding to each feature map provided in this specification. In the figure, A is a feature map in the image feature pyramid, the cube represents convolution, and the arrow on the cube represents the direction of convolution. In the figure, the feature map A is first convolved from top to bottom, then from bottom to top, then from left to right, and then from right to left, to obtain the convolution result B as the reconstructed feature corresponding to the feature map. The value of each pixel in Figure B is determined by the values ​​of the pixels around it.

[0079] Of course, when convolving the feature map, if up-down convolution is performed, the convolution kernel can also be a row vector, and if left-right convolution is performed, the convolution kernel can also be a column vector. The convolution method can also be to convolve the feature map with a convolution kernel in the positive direction, and each pixel after convolution is determined by the pixels adjacent to the pixel. The specific convolution kernel form and convolution direction can be set as needed and are not limited in this specification.

[0080] Furthermore, after determining the reconstruction codes corresponding to the feature maps, the server may determine the fusion results corresponding to the feature maps.

[0081] Specifically, the server may upsample the reconstructed code corresponding to each feature map, determine the upsampling result corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine the fusion result. Figure 4 shown.

[0082] Figure 4 This is a schematic diagram of determining the fusion result provided in this manual. In the figure, the image feature pyramid contains 5 feature maps, namely feature Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 and features Figure 5 , the sizes from large to small are characterized by Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 ,feature Figure 5 To determine the characteristics Figure 5 Taking the fusion result as an example, the server can determine the feature Figure 5 Reconstruction encoding: Features Figure 6 , and the features Figure 6 Upsampling to determine features Figure 7 , and then the features Figure 7 and features Figure 4 Perform fusion and determine the fusion results: features Figure 8 .

[0083] Of course, with the characteristics Figure 7 The fused feature map can also be a feature Figure 1 、 2 , 3, the specific fusion feature map can be based on the feature Figure 6 The size can be set as needed and is not limited in this specification.

[0084] Furthermore, in order to ensure that the image features of each feature map determined according to the fusion result are rich enough, when determining the fusion features, for each feature map, the server can upsample the reconstructed code of the feature map, and compare it with the image feature pyramid to which the feature map belongs, which has a size larger than that of the feature map. Figure 1 The server can fuse the feature maps of each level to determine the fusion result. For each image feature pyramid, the server can sort the sizes of each feature map from large to small, and divide each feature map into different levels according to the sorting. Figure 2 Take the image feature pyramid in as an example, the image feature pyramid includes five different levels.

[0085] S106: Taking the fusion results corresponding to the feature maps as input, inputting them into the recognition layer of the recognition model, and obtaining the recognition result of the training sample output by the recognition layer.

[0086] In one or more embodiments provided in this specification, after determining the fusion features corresponding to each image, the server may determine the recognition result of the training sample based on the fusion features.

[0087] Based on this, the server can input the fusion results corresponding to each feature map into the recognition layer of the recognition model to obtain the recognition result of the training sample output by the recognition layer.

[0088] Specifically, the server may take the fusion results corresponding to each feature map as input and input it into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to each feature map.

[0089] The server then inputs the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample. The recognition result can be the structure and position of the table in the training sample, or can be each recognition result determined according to a preset recognition result type.

[0090] Furthermore, the recognition result may include twenty-nine types, and the recognition layer is a twenty-nine-channel convolutional neural network layer, wherein five layers are used for semantic segmentation and twenty-four layers are used for pixel classification.

[0091] For each channel, the server can determine the channel's corresponding type and the corresponding recognition result. For example, the semantic segmentation results may include categories such as the location of the cell, the location of the table border, and the location of the cell border. Pixel classification mainly determines, for each pixel, whether the surrounding pixels are of the same category as the pixel, whether the pixel is an edge, whether it is inside a cell, or whether it is outside a table.

[0092] Of course, the specific number of channels and categories can be set as needed, and this manual does not impose any restrictions on this.

[0093] S108: Training the recognition model according to the recognition results and annotations of each training sample.

[0094] In one or more embodiments provided in this specification, after determining the recognition results and labels of each training sample, the server may train the recognition model based on the recognition results and labels of each training sample.

[0095] Specifically, when the recognition result is the structure and position of a table in the training sample, the server may determine the loss based on the position difference between the recognition result of the training sample and the corresponding pixel of the table in the annotation.

[0096] When the recognition results are determined according to the types of preset recognition results, the server can determine the labels of the training samples corresponding to the type for each preset type based on the labels of each training sample, and determine the loss based on the labels corresponding to the type and the recognition results, and determine the total loss as the sum of the losses of each type, and adjust the parameters of the model based on the total loss.

[0097] Taking the structure and position of a table in a training sample as an example of a recognition result, the server can determine the loss based on the difference between the position of the pixels belonging to the frame of the table in the recognition result of each training sample and the position of the pixels corresponding to the frame of the table in the annotation, and adjust the model parameters of the recognition model based on the loss.

[0098] based on Figure 1The training method of the recognition model determines several images containing tables as training samples. For each training sample, the label of the training sample is determined based on the structure and position of the table in the training sample. The training sample is input into the feature extraction layer of the recognition model to determine the image feature pyramid corresponding to the training sample. For each feature map in the image feature pyramid, the reconstruction code corresponding to the feature map is determined, and the reconstruction code corresponding to the feature map is upsampled and fused with other feature maps with a size larger than the feature map. The fusion results corresponding to each feature map are used as input and input into the recognition layer of the recognition model to obtain the recognition result of the training sample. This solution is based on the fusion of feature maps of different sizes to determine the recognition result of the training sample. The obtained image features are more comprehensive, and a rich amount of information is obtained at the moment of recognition of the captured image, with high efficiency.

[0099] based on Figure 1 The flow chart of the training method of the recognition model is provided in this specification. The structural diagram of the recognition model is provided in this specification. Figure 5 shown.

[0100] Figure 5 A schematic diagram of the structure of the recognition model provided in this specification shows that the recognition model includes a feature extraction layer and a recognition layer. The server can use an image containing a table as input, input it into the feature extraction layer, upsample and downsample the image, determine the upsampling result and downsampling result of the image, and then determine the upsampling result of the reconstructed code corresponding to the image. Each upsampling result is fused separately to obtain the fusion result corresponding to each feature map, and each fusion result is input into the recognition layer of the recognition model. After passing through the convolution sublayer and fusion sublayer of the recognition layer, different types of recognition results of the same size are output through different channels. The number of channels is related to the number of preset recognition results.

[0101] The specific number of channels can be set as needed and is not limited in this manual.

[0102] It should be noted that, in the process of inputting the fusion results into the recognition layer of the recognition model, the up-sampling result of the reconstructed code corresponding to the smallest feature map and the up-sampling result of the feature map with a size larger than the feature map can also be used. Figure 1 The reconstructed codes of the feature maps of each level are fused, and the fusion results are further upsampled to fuse the upsampled results with the reconstructed codes of the feature maps of larger sizes, until the reconstructed codes of the feature maps corresponding to each size are fused, and the fusion results are input into the recognition layer of the recognition model to determine the recognition results corresponding to the training samples.

[0103] based on Figure 1 In addition to the training method of the recognition model shown in FIG, this specification also provides a table recognition method, such as Figure 6shown.

[0104] Figure 6 The flowchart of the table recognition method provided in this manual includes:

[0105] S200: Determine an image containing a table.

[0106] S202: The image is input into a feature extraction layer of a pre-trained recognition model, and the image is upsampled and downsampled respectively to determine an image feature pyramid corresponding to the image.

[0107] S204: For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, and the reconstructed code corresponding to the feature map is upsampled and fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result.

[0108] S206: Taking the fusion results corresponding to each feature map as input, inputting them into the recognition layer of the recognition model, obtaining the recognition result of the image output by the recognition layer, and determining the structure and position of the table in the image based on the recognition result of the image.

[0109] The above steps S200-S206 may refer to the above steps S100-S106, and will not be described in detail in this specification.

[0110] In one or more embodiments provided in this specification, the table recognition method can be applied to a scenario of recognizing an image containing a table and determining the frame and position of the table in the image.

[0111] The table recognition method is executed by a server. The server that executes the table recognition method and the server that trains the recognition model can be the same server or different servers.

[0112] The server may input an image containing a table into a pre-trained recognition model, obtain recognition results output by the recognition model, and after determining the recognition results of the image, determine the pixels corresponding to the structure of the table in the image based on the recognition results, and then determine the position and structure of the table based on the positions of the pixels. The determined position of the table may be the position of the table border, or may be the position of each cell contained in the table.

[0113] The above is a method for training a recognition model provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding training device for a recognition model, such as Figure 7 shown.

[0114] Figure 7 The training device for the recognition model provided in this manual includes:

[0115] The sample determination module 300 is configured to determine a number of images containing tables as training samples, and determine a label for each training sample based on the structure and position of the table in the training sample.

[0116] The sampling module 302 is configured to input the training sample as an input to a feature extraction layer of a recognition model to be trained, perform upsampling and downsampling on the training sample, and determine an image feature pyramid corresponding to the training sample.

[0117] The fusion module 304 is used to perform convolution in different directions on each feature map in the image feature pyramid, determine the reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine the fusion result.

[0118] The recognition module 306 is configured to input the fusion results corresponding to the feature maps into the recognition layer of the recognition model, and obtain the recognition result of the training sample output by the recognition layer.

[0119] The training module 308 is used to train the recognition model according to the recognition results and annotations of each training sample.

[0120] Optionally, the recognition results include multiple types, and the sample determination module 300 is used to determine, for each training sample, the labels corresponding to each type of the training sample based on the preset recognition result type and the label of the training sample, determine the loss based on the recognition results and labels of each type corresponding to each training sample, and adjust the model parameters of the recognition model based on the loss.

[0121] Optionally, the recognition module 306 is used to take the fusion results corresponding to each feature map as input, input them into the convolution sublayer of the recognition layer of the recognition model, determine the fusion results of the same size corresponding to each feature map, input the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model, and obtain the recognition result output by the training sample.

[0122] Optionally, the fusion module 304 is configured to upsample the reconstructed code corresponding to each feature map in the image feature pyramid, and fuse it with a feature map one level larger than the size of the feature map to determine a fusion result.

[0123] Optionally, the recognition module 306 is used to input the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model, determine the fusion features corresponding to the training sample, and obtain the recognition results of the training sample output by each preset channel according to the preset recognition result type and the fusion features.

[0124] The above is a table recognition method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding table recognition device, such as Figure 8 shown.

[0125] Figure 8 The table recognition device provided for this manual includes:

[0126] The first determining module 400 is configured to determine an image containing a table.

[0127] The second determination module 402 is configured to take the image as input and input it into a feature extraction layer of a pre-trained recognition model, perform upsampling and downsampling on the image, and determine an image feature pyramid corresponding to the image.

[0128] The fusion module 404 is used to perform convolution in different directions on each feature map in the image feature pyramid, determine the reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine the fusion result.

[0129] The third determination module 406 is used to input the fusion results corresponding to each feature map into the recognition layer of the recognition model, obtain the recognition result of the image output by the recognition layer, and determine the structure and position of the table in the image based on the recognition result of the image.

[0130] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 The training method of the recognition model provided or Figure 6 Provided table recognition method.

[0131] This manual also provides Figure 9 The schematic structure diagram of the electronic device shown in FIG. Figure 9 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The training method of the recognition model or Figure 6Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0133] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0134] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0135] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0136] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0144] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0146] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0147] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for training a recognition model, characterized in that: include: Determine a number of images containing tables as training samples, and for each training sample, determine a label for the training sample based on the structure and position of the table in the training sample; The training sample is input into the feature extraction layer of the recognition model to be trained, and the training sample is upsampled and downsampled respectively to determine the image feature pyramid corresponding to the training sample; For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, upsampling the reconstructed code corresponding to the feature map is performed, and the reconstructed code is fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result; The fusion results corresponding to each feature map are input into the recognition layer of the recognition model to obtain the recognition result of the training sample output by the recognition layer; Training the recognition model based on the recognition results and annotations of each training sample; The fusion results corresponding to each feature map are used as input to the recognition layer of the recognition model to obtain the recognition result of the training sample output by the recognition layer, specifically including: The fusion results corresponding to the feature maps are input into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to the feature maps; The fusion results of the same size corresponding to each feature map are input into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample.

2. The method according to claim 1, wherein The recognition results include multiple types; The recognition model is trained based on the recognition results and annotations of each training sample, specifically including: For each training sample, determine the label corresponding to each type of the training sample according to the preset recognition result type and the label of the training sample; According to the recognition results and labels of the various types corresponding to the training samples, a loss is determined, and model parameters of the recognition model are adjusted according to the loss.

3. The method according to claim 1, wherein The reconstructed code corresponding to the feature map is up-sampled, and the code is fused with other feature maps in the image feature pyramid whose size is larger than that of the feature map to determine a fusion result. Specifically, the method includes: for each feature map in the image feature pyramid, the reconstructed code corresponding to the feature map is up-sampled, and the code is fused with a feature map whose size is one level larger than that of the feature map to determine a fusion result.

4. The method according to claim 1, wherein The recognition results include twenty-nine types, and the recognition layer is a twenty-nine-channel convolutional neural network layer, of which five layers are used for semantic segmentation and twenty-four layers are used for pixel classification; Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample specifically includes: Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to determine the fusion features corresponding to the training sample; According to the preset recognition result type and the fusion feature, the recognition result of the training sample output by each preset channel is obtained.

5. A table recognition method, characterized in that: include: Identify the image containing the table; Taking the image as input, inputting it into the feature extraction layer of a pre-trained recognition model, upsampling and downsampling the image respectively, and determining an image feature pyramid corresponding to the image; For each feature map in the image feature pyramid, convolution is performed on the feature map in different directions to determine a reconstructed code corresponding to the feature map, upsampling the reconstructed code corresponding to the feature map is performed, and the reconstructed code is fused with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result; Using the fusion results corresponding to each feature map as input, inputting them into the recognition layer of the recognition model, obtaining the recognition result of the image output by the recognition layer, and determining the structure and position of the table in the image based on the recognition result of the image; The fusion results corresponding to each feature map are used as input to the recognition layer of the recognition model to obtain the recognition result of the image output by the recognition layer, specifically including: The fusion results corresponding to the feature maps are input into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to the feature maps; The fusion results of the same size corresponding to each feature map are input into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result of the image.

6. A training device for a recognition model, characterized in that: include: a sample determination module, configured to determine a number of images containing tables as training samples, and for each training sample, determine a label for the training sample based on the structure and position of the table in the training sample; A sampling module is used to input the training sample as input to the feature extraction layer of the recognition model to be trained, upsample and downsample the training sample respectively, and determine the image feature pyramid corresponding to the training sample; a fusion module, configured to perform convolution in different directions on each feature map in the image feature pyramid, determine a reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, and fuse it with other feature maps in the image feature pyramid that are larger than the feature map to determine a fusion result; The recognition module is used to input the fusion results corresponding to each feature map as input to the recognition layer of the recognition model, and obtain the recognition results of the training sample output by the recognition layer. The recognition module is used to input the fusion results corresponding to each feature map as input to the recognition layer of the recognition model, and obtain the recognition results of the training sample output by the recognition layer. Specifically, it includes: The fusion results corresponding to the feature maps are input into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to the feature maps; Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result output by the training sample; The training module is used to train the recognition model according to the recognition results and annotations of each training sample.

7. A table recognition device, characterized in that: include: A first determining module, configured to determine an image containing a table; A second determination module is configured to take the image as input and input it into a feature extraction layer of a pre-trained recognition model, perform upsampling and downsampling on the image, and determine an image feature pyramid corresponding to the image; A fusion module is configured to perform convolution in different directions on each feature map in the image feature pyramid, determine a reconstructed code corresponding to the feature map, upsample the reconstructed code corresponding to the feature map, fuse it with other feature maps in the image feature pyramid that are larger than the feature map, determine a fusion result, and input the fusion result corresponding to each feature map as input to the recognition layer of the recognition model to obtain a recognition result of the image output by the recognition layer. Specifically, the fusion module includes: The fusion results corresponding to the feature maps are input into the convolution sublayer of the recognition layer of the recognition model to determine the fusion results of the same size corresponding to the feature maps; Inputting the fusion results of the same size corresponding to each feature map into the fusion sublayer of the recognition layer of the recognition model to obtain the recognition result of the image; The third determination module is used to input the fusion results corresponding to each feature map into the recognition layer of the recognition model, obtain the recognition result of the image output by the recognition layer, and determine the structure and position of the table in the image based on the recognition result of the image.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 or claim 5 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 or claim 5 is implemented.

Images