Table detection method, device, electronic device and storage medium

Abstract

The present invention provides a table detection method, device, electronic device and storage medium, and relates to the field of image recognition technology. The table detection method includes: firstly, performing model detection on a first image to extract text position information and generate a detection result; then binarizing the detection result to generate a binary image; then channel splicing the binary image with the first image to generate a second image, and finally sending the second image to a detection network for detection to generate a table detection result. Such a table detection method can make the position information and arrangement relationship of text lines in the image be explicitly used by the neural network, effectively improve the effect of OCR table detection, make the table detection result more accurate, and pave the way for subsequent table analysis.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image recognition, and in particular to a table detection method, device, electronic device and storage medium. Background Art

[0002] In current daily business activities, we use a large number of documents and forms every day, and form documents are also widely used in various fields. Optical Character Recognition (OCR) form analysis plays a very important role in text images such as forms, bills, and financial statements.

[0003] Table detection is often the first step in table analysis. Table detection is to locate the area where the table is located in the text image. In the prior art, the main method for table detection is: through the image to be tested, the table line elements therein are extracted to obtain a preprocessed table line image, and the target table line image is obtained based on the intersection of the table lines, thereby obtaining a table detection result.

[0004] However, in practical applications, determining table detection information only by table lines while ignoring the position information and row-column relationship of text rows in the text image will result in low accuracy of table detection results, which is not conducive to subsequent table analysis. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present application is to propose a method, device, electronic device and storage medium for table detection, which generates a binary image by using a pre-acquired text line detection model, and then performs channel splicing with the original image and sends it into a table detection network, thereby improving the effect of OCR table detection and paving the way for subsequent table analysis.

[0006] In a first aspect, an embodiment of the present invention provides a table detection method, comprising:

[0007] Performing model detection on the first image to extract text position information and generate a detection result;

[0008] Binarizing the detection result to generate a binary image;

[0009] Perform channel splicing on the binary image and the first image to generate a second image;

[0010] The second image is sent to the detection network for detection to generate a table of detection results.

[0011] Preferably, performing model detection on the first image to extract text position information includes:

[0012] The model includes a pre-acquired text line detection model;

[0013] The text position information includes position information of text lines and row-column relationship information.

[0014] Preferably, generating the detection result includes:

[0015] The detection result is a result of text line detection, and the position information and row-column relationship of the text line in the first image are extracted.

[0016] Preferably, binarizing the detection result to generate a binary image includes:

[0017] The binarization process is to set a binarization threshold according to the data information of the detection result, wherein the data information is the position information of the text line;

[0018] The detection result is binarized according to the set threshold to generate a binary image.

[0019] Preferably, performing channel stitching on the binary image and the first image to generate a second image comprises:

[0020] On the basis of the first image including three channels of RGB, the binary image is taken as an additional channel to generate a second image by splicing.

[0021] Preferably, sending the second image to a detection network for detection includes:

[0022] The detection network is a table detection network using a deep learning algorithm based on regression or segmentation.

[0023] In a second aspect, an embodiment of the present application shows a table detection device, including a model detection module, a binary image module, a channel splicing module, and a detection network module, wherein:

[0024] The model detection module is used to perform model detection on the first image to extract text position information and generate a detection result;

[0025] The binarization module is used to perform binarization processing on the detection result to obtain a binary image;

[0026] The channel stitching module is used to perform channel stitching on the binary image and the first image to generate a second image;

[0027] The detection network module is used to send the second image into the detection network for detection and generate a table of detection results.

[0028] In a third aspect, an embodiment of the present invention shows an electronic device, including:

[0029] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory has instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the table detection method as described in any one of the first aspects.

[0030] In a fourth aspect, an embodiment of the present invention shows a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the table detection method described in any one of the first aspects is implemented.

[0031] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0032] In an embodiment of the present invention, the first acquired image is first subjected to model detection, and text position information is extracted through model detection to generate a detection result. The detection result is then binarized to generate a binary image. The binary image and the first image are then channel-joined to generate a second image. Finally, the second image is placed in a detection network for table detection to obtain a table detection result. In this way, the position information and arrangement relationship of the text rows can be explicitly used by the table detection neural network, which can effectively improve the effect of table detection and pave the way for subsequent table analysis.

[0033] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0035] Figure 1 is a flowchart of a table detection method embodiment provided by an embodiment of the present invention;

[0036] Figure 2 is a structural block diagram of an embodiment of a table detection device provided by an embodiment of the present invention;

[0037] Figure 3 It is a schematic diagram of the structure of an electronic device embodiment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0039] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0040] Figure 1 1 is a flow chart of table detection in an embodiment of the present invention. Figure 1 As shown, the process of table detection includes the following steps:

[0041] Step 101: Perform model detection on the first image to extract text position information and generate a detection result;

[0042] The first image may be an image obtained by photographing or scanning, or may be an image obtained by converting the format of a document, table, presentation, etc. These images may contain text lines of different sizes, numbers, positions, and arrangements. The number of text lines is not required, and the content of the text lines may be in Chinese, English, or other languages, which is not limited in the present invention.

[0043] In the embodiment of the present invention, model detection refers to detection through a text line detection model, wherein the text line detection model pre-selects table detection data as a training set and performs text line detection on the training set samples.

[0044] For example, the table detection is performed on the text images uploaded by the customer. The samples include 100,000 open source table detection data and 100,000 self-annotated table detection data of the company. 95,000 open source table detection data and 95,000 self-annotated data of the company are randomly selected as the training set, and the remaining 10,000 samples are used as the test set. The text image has been tilted. For example, the tilt correction can be performed by using the Hough straight line method to detect straight lines on the binary image, arranging all the detected straight lines according to their length, taking the first 50 largest straight lines, and if there are less than 50, taking all the straight lines, performing slope statistics, and taking the angle corresponding to the slope with the most occurrences as the image tilt angle, constructing the corresponding change matrix to correct the image, and using two-point annotation when annotating, that is, annotating the upper left corner and lower right corner of the table area. Then, the text line detection is performed on the training set samples and the test set samples.

[0045] In the embodiment of the present invention, the text line detection is to locate the position of the text line in the text image by using commonly used algorithms such as psenet and dbnet.

[0046] Among them, the PSENET algorithm is based on the semantic segmentation method, then reduces the size of the original text line, and finally gradually expands the size to locate the position of the text line.

[0047] In the embodiment of the present invention, the text position information in the first image is initially extracted, wherein the text position information includes the position information of the text row and the information of the row-column relationship. For example, the text information is divided by the minimum circumscribed quadrilateral, and the generated detection result is that the text part is generally marked in the manner of the minimum circumscribed quadrilateral, and all the text parts in the image are marked, and the number is determined according to the position of the text, and the present invention does not limit the number and position of this part.

[0048] Step 102: Binarize the detection result to generate a binary image.

[0049] In the embodiment of the present invention, the binary image refers to an image in which each pixel has only two possible values ​​or grayscale states, such as a black and white image, a monochrome image, etc. In a preferred embodiment of the present invention, the binary image can be an image in which the value is 1 where there is text and the value is 0 elsewhere.

[0050] In the embodiment of the present invention, the detection result is binarized, and the binarization is performed by setting a threshold value for binarizing the data information of the detection result, wherein the data information is the position information of the text line. The binarization is performed according to the threshold value set according to the text position information of the text line in the detection result to generate a binary image, and the size of the binary image is consistent with the first image.

[0051] By detecting the text content in the result, for example, according to the four coordinate points of the minimum circumscribed quadrilateral, namely (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4), the text content inside is filled in, and the places with text are filled with 1, and the non-text areas are filled with 0, so as to obtain a binary image.

[0052] Step 103: perform channel splicing on the binary image and the first image to generate a second image.

[0053] In an embodiment of the present invention, the RGB three channels in the first image are spliced ​​with the binary image to generate a new four-channel input, wherein the four channels are black and white channels. The purpose of channel splicing in the present invention is to reduce the distortion of the image during the processing and improve the accuracy of the table detection results.

[0054] Step 104: Detect the second image to obtain a table detection result.

[0055] In an embodiment of the present invention, the detection method is to obtain the detection result in a table detection network based on a deep learning algorithm of regression or segmentation.

[0056] Among them, the regression-based table detection algorithm can select the YOLO series, SDD series, FASTER-RCNN series, etc., and regress the upper left and lower right fixed points of the table, which is consistent with the idea of ​​generalized target detection; the segmentation-based table detection algorithm can select common algorithms such as UNET and FCN, take the table area as the segmentation target, and then use the post-processing algorithm to get the position of the table.

[0057] For example, the contraction path in the Unet table detection algorithm uses the convolution pooling component of the traditional convolutional neural network, in which the number of channels is doubled after one downsampling. The expansion path consists of a 2*2 deconvolution, and the output channel of the deconvolution is half of the original number of channels, which is then connected in series with the output (feature map) of the convolution layer in the original convolutional network to obtain a feature map with the same number of channels as the original, and then passes through two convolutions of size 3*3 and the linear rectifier function (Rectified Linear Unit, ReLU). In the last layer, the desired target type is obtained through the convolution of the convolution kernel size of 1*1.

[0058] In this embodiment, the first image is first placed in a pre-acquired text row detection model for detection to obtain a detection result containing the position information and row-column relationship of the text row, and then the detection result is binarized based on the set threshold to obtain a binary image, and then the binary image is spliced ​​with the RGB three channels of the first image to obtain a new four-channel input to generate a second image, and finally the second image is sent to the table detection neural network to obtain a table detection result. Through this method, the final table detection neural network can effectively make explicit use of the position information and row-column relationship of the text row in the first image, making the positioning of the table detection more accurate, thereby achieving a more accurate table detection result. This method can effectively solve the problem of inaccurate detection results in table detection.

[0059] In order to implement the above table detection process, another embodiment of the present invention discloses a table detection device.

[0060] Figure 2 Schematic diagram of the structure of the table detection device in the embodiment of the present invention. Figure 2 As shown, the table detection device includes a detection model module 201, a binary image module 202, a channel splicing module 203, and a detection network module 204.

[0061] The detection model module 201 is used to perform model detection on the first image to extract text position information and generate a detection result.

[0062] Among them, the model can form a text line detection model through multiple text line detection results obtained in advance.

[0063] The binarization module 202 is used to perform binarization processing on the detection result to obtain a binary image.

[0064] In this embodiment, the binarization module binarizes the detection result according to a threshold value set for the detection result, ie, the position information of the text line.

[0065] The channel stitching module 203 is used to perform channel stitching on the binary image and the first image to generate a second image.

[0066] In this embodiment, when performing channel stitching, the RGB channels of the first image are first split to obtain three channels, and then stitched with the new four channels formed by the binary image to obtain the second image. The purpose is to prevent the image from being distorted in the processing of the previous parts, and the problem of inconsistent position information and row and column relationships of the text lines in the first image. This can effectively avoid the occurrence of such problems and make the final table test results more accurate.

[0067] The detection network module 204 is used to detect the second image and obtain a detection result, wherein the detection network used is a detection performed in a table detection network based on a deep learning algorithm based on regression or segmentation.

[0068] The algorithm is as described above and will not be described in detail here.

[0069] By adopting the above-mentioned embodiment of the present invention, due to the use of binarization processing and channel splicing, it is possible to ensure that the position information and row-column relationship of the text in the table detection are accurate, thereby improving the efficiency of the detection result.

[0070] Another embodiment of the present invention relates to an electronic device, such as Figure 3 As shown, including:

[0071] At least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute the table detection method described in the embodiment of the present invention.

[0072] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0073] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0074] This embodiment also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned table detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0075] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0076] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned table detection embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0077] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0078] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention.

[0079] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A table detection method, characterized in that: The following steps are involved: Performing model detection on the first image to extract text position information and generate a detection result; The model includes a pre-acquired text line detection model; The text position information includes position information of text lines and row-column relationship information; Binarizing the detection result to generate a binary image includes: The binarization process is to set a binarization threshold according to the data information of the detection result, wherein the data information is the position information of the text line; The detection result is binarized according to the set threshold value to generate a binary image; Perform channel splicing on the binary image and the first image to generate a second image; The second image is sent to the detection network for detection to generate a table of detection results.

2. The table detection method according to claim 1, characterized in that: The generating of the detection result comprises: The detection result is a result of text line detection, and the position information and row-column relationship of the text line in the first image are extracted.

3. The table detection method according to claim 1, characterized in that: The binary image and the first image are channel-joined to generate a second image, comprising: On the basis of the first image including three channels of RGB, the binary image is taken as an additional channel to generate a second image by splicing.

4. The table detection method according to claim 1, characterized in that: The sending the second image into a detection network for detection includes: The detection network is a table detection network using a deep learning algorithm based on regression or segmentation.

5. A table detection device, characterized in that: Including model detection module, binarization module, channel splicing module, detection network module, including: The model detection module is used to perform model detection on the first image to extract text position information and obtain a detection result; The model includes a pre-acquired text line detection model; The text position information includes the position information of the text line and the row-column relationship information; The binarization module is used to perform binarization processing on the detection result to obtain a binary image; The binarization process is to set the binarization threshold according to the data information of the detection result, wherein the data information is the position information of the text line; The detection result is binarized according to the set threshold value to generate a binary image; The channel stitching module is used to perform channel stitching on the binary image and the first image to generate a second image; The detection network module is used to send the second image into the detection network for detection and generate a table of detection results.

6. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the table detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the table detection method according to any one of claims 1 to 4 is implemented.

Images