Table recognition method, system, intelligent terminal and computer-readable storage medium

Abstract

The present invention discloses a table recognition method, system, intelligent terminal and computer-readable storage medium, wherein the method comprises: obtaining an image to be processed, performing character recognition on the image to be processed, and obtaining text information in the image to be processed; obtaining image features of the image to be processed, and performing tilt angle correction on the image to be processed according to the image features and text information to obtain a corrected image, and performing tilt angle correction on the text information to obtain text correction information; obtaining each denoised table area in the corrected image according to a pre-trained table area detection model, cutting and splicing the corrected image according to the denoised table area to obtain a target denoised image, and obtaining text denoised information according to the target denoised image and text correction information; performing table structure recognition according to the text denoised information and the target denoised image and obtaining multiple target tables. Compared with the prior art, the present invention is conducive to improving the efficiency of table recognition.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a table recognition method, system, intelligent terminal and computer-readable storage medium. Background Art

[0002] At present, the application of tables is becoming more and more extensive, and more and more tables need to be processed by users. However, it is often impossible to directly extract, edit and use tables in images or pictures, and they rely on manual recognition and extraction.

[0003] In the prior art, the table in the image needs to be manually identified, and a corresponding table is manually created on the electronic device according to the table in the image. The problem of the prior art is that the solution of manually identifying the table in the image is not conducive to improving the efficiency of table recognition.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a table recognition method, system, intelligent terminal and computer-readable storage medium, aiming to solve the problem that the solution of manually recognizing tables in images in the prior art is not conducive to improving the efficiency of table recognition.

[0006] In order to achieve the above object, the present invention provides a table recognition method in a first aspect, wherein the method comprises:

[0007] Acquire an image to be processed, perform character recognition on the image to be processed, and acquire text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates;

[0008] Acquire the image features of the image to be processed, and perform tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and perform tilt angle correction on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, and the length of the target straight line is greater than a preset length threshold, and the text correction information includes text correction content and text box correction coordinates;

[0009] According to the pre-trained table area detection model, each denoised table area in the above-mentioned corrected image is respectively obtained, and according to the above-mentioned denoised table area, the above-mentioned corrected image is cropped and spliced ​​to obtain a target denoised image, and the text denoising information is obtained according to the above-mentioned target denoised image and the above-mentioned text correction information, wherein the above-mentioned target denoised image only includes all the above-mentioned denoised table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content;

[0010] Table structure recognition is performed based on the text denoising information and the target denoising image to obtain multiple target tables.

[0011] Optionally, the acquiring of the image features of the image to be processed, and performing tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and performing tilt angle correction on the text information to obtain text correction information, includes:

[0012] Performing Hough transform on the image to be processed to obtain at least one long horizontal straight line in the image to be processed, wherein the length of the long horizontal straight line is greater than the length threshold, and the length threshold is half of the width of the image to be processed;

[0013] The longest of all the above mentioned horizontal long straight lines is taken as the above mentioned target straight line;

[0014] According to the target straight line and the text information, the image to be processed is corrected for an inclination angle to obtain a corrected image, and the text information is corrected for an inclination angle to obtain text correction information.

[0015] Optionally, the step of performing tilt angle correction on the image to be processed according to the target straight line and the text information to obtain a corrected image, and performing tilt angle correction on the text information to obtain text correction information, includes:

[0016] The first image tilt angle is calculated based on the length of the text content and the text box coordinates, and the second image tilt angle is calculated based on the target straight line.

[0017] Obtaining a target image tilt angle according to the first image tilt angle and the second image tilt angle, wherein the target image tilt angle is a maximum value of a first absolute value and a second absolute value, the first absolute value is an absolute value of the first image tilt angle, and the second absolute value is an absolute value of the second image tilt angle;

[0018] Rotating the image to be processed according to the tilt angle of the target image to obtain a corrected image;

[0019] According to the inclination angle of the target image, the text box coordinates in the text information are inclination-corrected to obtain text box correction coordinates, and the text content corresponding to each of the text box correction coordinates is used as the text correction content in the text correction information to construct the text correction information.

[0020] Optionally, the above-mentioned obtaining each denoised table area in the above-mentioned corrected image according to the pre-trained table area detection model, cropping and splicing the above-mentioned corrected image according to the above-mentioned denoised table area to obtain a target denoised image, and obtaining text denoising information according to the above-mentioned target denoised image and the above-mentioned text correction information, including:

[0021] Acquire each table area in the above-mentioned rectified image according to the above-mentioned pre-trained table area detection model;

[0022] Acquire each table cell frame area in the above-mentioned corrected image according to a pre-trained table cell frame detection model;

[0023] Acquire the noise reduction table area according to the table area and the table cell frame area;

[0024] According to the denoising table area, the corrected image is cropped and spliced ​​to obtain a target denoised image;

[0025] The text correction content in the text correction information that belongs to the noise reduction table area is used as the text noise reduction content, and the text box noise reduction coordinates corresponding to each of the text noise reduction contents are obtained according to the target noise reduction image, the corrected image and the text box correction coordinates.

[0026] Optionally, the above-mentioned performing table structure recognition according to the above-mentioned text denoising information and the above-mentioned target denoising image and obtaining multiple target tables includes:

[0027] Acquire the text denoising content in each table unit frame area in the above target denoising image, respectively perform row and column splicing adjustments on the text denoising content in each table unit frame area to obtain each text splicing content, and obtain the text frame splicing coordinates corresponding to each text splicing content, and construct text splicing information composed of the above text splicing content and the above text frame splicing coordinates;

[0028] The target table is obtained according to the above text splicing information, the above target denoised image and the pre-trained table structure recognition model.

[0029] Optionally, the step of obtaining a target table according to the text concatenation information, the target denoised image and a pre-trained table structure recognition model includes:

[0030] The target table is obtained by processing through preset loop processing steps, wherein the i-th loop processing step includes:

[0031] Inputting the input data into the table structure recognition model to perform table recognition and obtain the to-be-processed information of the recognized reconstructed table, wherein the to-be-processed information includes the cell row and column information, cell coordinates and cell text content of the reconstructed table;

[0032] Correct the text content of each cell in the reconstructed table according to the preset knowledge base to obtain a first corrected table;

[0033] Acquire the form confirmation information input by the target object, and when the form confirmation information indicates successful recognition, use the first revised form as the target form, output the target form and the loop ends;

[0034] When the table confirmation information indicates that the recognition is incorrect, the table modification instruction data input by the target object is obtained, the first modified table is modified according to the table modification instruction data to obtain a second modified table, and a modified image, cell text content and cell coordinates corresponding to the second modified table are obtained, and the modified image, cell text content and cell coordinates corresponding to the second modified table are used as input data of the table structure recognition model and the i+1th cycle is performed;

[0035] Wherein, i is an integer greater than or equal to 1, and the input data of the above table structure recognition model in the first cycle includes the above text splicing information and the above target denoised image.

[0036] Optionally, the above-mentioned correcting the text content of each cell in the above-mentioned reconstructed table according to a preset knowledge base to obtain a first corrected table includes:

[0037] Obtain a preset knowledge base, preset tabulation rules, and pre-trained language models;

[0038] Extract keywords from the above knowledge base, and obtain row and column association information corresponding to each of the above keywords according to the above tabulation rules;

[0039] Correcting the text content of each cell of the reconstructed table according to the keywords and the row-column association information to obtain a first corrected text;

[0040] Acquire a first word confidence corresponding to the first revised text according to the input data and the first revised text, and revise the first revised text according to the first word confidence and the language model to obtain a second revised text;

[0041] Obtaining the second word confidence corresponding to the second revised text according to the input data, performing glyph matching and correction on the second revised text according to the second word confidence, the keyword and a preset glyph similarity threshold, to obtain a third revised text;

[0042] The first correction table is generated according to the cell row and column information, the cell coordinates and the third correction text.

[0043] A second aspect of the present invention provides a table recognition system, wherein the system comprises:

[0044] The image processing module is used to obtain the image to be processed, perform character recognition on the image to be processed, and obtain text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates;

[0045] A correction module, used for acquiring image features of the image to be processed, and performing tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and performing tilt angle correction on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, and the length of the target straight line is greater than a preset length threshold, and the text correction information includes text correction content and text box correction coordinates;

[0046] A denoising module, for respectively acquiring each denoising table area in the corrected image according to a pre-trained table area detection model, cropping and splicing the corrected image according to the denoising table area to obtain a target denoised image, and obtaining text denoising information according to the target denoised image and the text correction information, wherein the target denoised image only includes all the denoising table areas, and the text denoising information includes text denoising content within the denoising table area and text box denoising coordinates corresponding to the text denoising content;

[0047] The table recognition module is used to perform table structure recognition based on the text denoising information and the target denoising image and obtain multiple target tables.

[0048] A third aspect of the present invention provides an intelligent terminal, comprising a memory, a processor, and a table recognition program stored in the memory and executable on the processor, wherein the table recognition program implements any one of the steps of the table recognition method when executed by the processor.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium, on which a table recognition program is stored, and when the table recognition program is executed by a processor, the steps of any one of the table recognition methods are implemented.

[0050] As can be seen from the above, in the scheme of the present invention, an image to be processed is obtained, character recognition is performed on the image to be processed, and text information in the image to be processed is obtained, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates; image features of the image to be processed are obtained, and according to the image features and the text information, the image to be processed is subjected to tilt angle correction to obtain a corrected image, and the text information is subjected to tilt angle correction to obtain text correction information, wherein the image features include a target straight line in the image to be processed, and the length of the target straight line is greater than a preset length threshold, and the text correction information includes text Correction content and text box correction coordinates; obtaining each denoising table area in the above-mentioned corrected image according to a pre-trained table area detection model, cropping and splicing the above-mentioned corrected image according to the above-mentioned denoising table area to obtain a target denoising image, and obtaining text denoising information according to the above-mentioned target denoising image and the above-mentioned text correction information, wherein the above-mentioned target denoising image only includes all the above-mentioned denoising table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content; performing table structure recognition according to the above-mentioned text denoising information and the above-mentioned target denoising image and obtaining multiple target tables.

[0051] Compared with the prior art which requires manual recognition of tables in images and relies on manual production of corresponding tables on electronic devices, the present invention can automatically recognize tables in images to be processed, obtain text information in the image to be processed, perform tilt angle correction and noise reduction on the image to be processed and the text information obtained by recognition, and finally obtain the target table in the image to be processed, without the need for manual recognition of the contents of the image one by one, which is conducive to improving the efficiency of table recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 It is a flowchart of a table recognition method provided by an embodiment of the present invention;

[0054] Figure 2 Embodiment of the present invention Figure 1 Specific flow diagram of step S200;

[0055] Figure 3 It is a schematic diagram of selecting and deleting a table through an interactive interface provided by an embodiment of the present invention;

[0056] Figure 4 It is a specific flow chart of a table recognition method provided by an embodiment of the present invention;

[0057] Figure 5 It is a structural schematic diagram of a table recognition system provided by an embodiment of the present invention;

[0058] Figure 6 is a structural diagram of another table recognition system provided by an embodiment of the present invention;

[0059] Figure 7 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0061] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0062] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0063] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0064] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0065] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0067] With the development of science and technology and the advent of the digital age, the development of computer technology has received more and more attention, computer computing power has been continuously improved, and artificial intelligence technology has also been further developed. With the development of artificial intelligence technology, the technology of optical character recognition (OCR) has also been greatly improved, and its application has become more and more extensive. However, in complex scenarios, it also faces higher challenges.

[0068] At present, the application of forms is becoming more and more extensive, and more and more forms need to be processed by users. The demand for electronic scanned documents in various industries is increasing, and with it comes more and more complex application scenarios. For example, in a large enterprise, there will be different types of invoices, forms, reports, business contracts and other documents or images (pictures) with different structures, and the layout information needs to be analyzed, and the table information in the layout is particularly important. At the same time, the table structure is ever-changing, and it is difficult to adapt to most needs using simple template matching. Therefore, it is often impossible to directly extract, edit and use tables in images, and they rely on manual recognition and extraction.

[0069] In the prior art, tables in documents or images need to be manually identified, and a corresponding table is manually created on an electronic device based on the table in the image. The problem with the prior art is that the solution of manually identifying tables in images is not conducive to improving the efficiency of table recognition, and requires a large amount of human resources, which is not conducive to saving human resources.

[0070] In order to solve at least one of the above-mentioned problems, in the scheme of the present invention, an image to be processed is obtained, character recognition is performed on the image to be processed, and text information in the image to be processed is obtained, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates; image features of the image to be processed are obtained, and according to the image features and the text information, a tilt angle correction is performed on the image to be processed to obtain a corrected image, and a tilt angle correction is performed on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, the length of the target straight line is greater than a preset length threshold, and the text correction The positive information includes text correction content and text box correction coordinates; each denoising table area in the above-mentioned corrected image is obtained respectively according to a pre-trained table area detection model, the above-mentioned corrected image is cropped and spliced ​​according to the above-mentioned denoising table area to obtain a target denoising image, and text denoising information is obtained according to the above-mentioned target denoising image and the above-mentioned text correction information, wherein the above-mentioned target denoising image only includes all the above-mentioned denoising table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content; table structure recognition is performed according to the above-mentioned text denoising information and the above-mentioned target denoising image to obtain multiple target tables.

[0071] Compared with the prior art which requires manual recognition of tables in images and relies on manual production of corresponding tables on electronic devices, the present invention can automatically recognize tables in images to be processed, obtain text information in the image to be processed, perform tilt angle correction and noise reduction on the image to be processed and the text information obtained by recognition, and finally obtain the target table in the image to be processed, without the need for manual recognition of the contents of the image one by one, which is beneficial to improving the efficiency of table recognition and saving human resources.

[0072] Exemplary Methods

[0073] like Figure 1 As shown, an embodiment of the present invention provides a table recognition method. Specifically, the method includes the following steps:

[0074] Step S100, obtaining an image to be processed, performing character recognition on the image to be processed, and obtaining text information in the image to be processed.

[0075] The image to be processed includes at least one table, and the text information includes text content and text box coordinates.

[0076] The above-mentioned image to be processed is an image (or picture) that needs to be recognized by a table. In this embodiment, OCR recognition is performed on the above-mentioned image to be processed to obtain the text information therein. Specifically, the text content at each position therein and the text box coordinates corresponding to each text content can be obtained by recognizing the image to be processed. The above-mentioned text content corresponds to the above-mentioned text box coordinates one by one. It should be noted that one text box coordinate corresponds to the coordinates of the four orders of the text box.

[0077] In this embodiment, the coordinate system corresponding to the above text box coordinates can be a pre-established world coordinate system. The coordinate system does not change during the process of tilt angle correction or denoising of the image, so there is no need to repeatedly establish the coordinate system, reducing the complexity of the calculation process.

[0078] Step S200, obtaining the image features of the image to be processed, and performing tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and performing tilt angle correction on the text information to obtain text correction information.

[0079] Among them, the above-mentioned image features include the target straight line in the above-mentioned image to be processed, the length of the above-mentioned target straight line is greater than a preset length threshold, and the above-mentioned text correction information includes text correction content and text box correction coordinates.

[0080] It should be noted that in actual use, the direction of the image to be processed may be tilted when it is input, causing the table therein to be tilted, or the table therein is tilted when the image to be processed is captured. Therefore, it is necessary to correct the tilt angle of the above-mentioned image to be processed according to the table.

[0081] Specifically, in this embodiment, the obtained text box coordinates, text content and the original image to be processed are used as inputs in the image correction process, and the tilted image to be processed is corrected to obtain a corrected image.

[0082] In this embodiment, Figure 2 As shown, the above step S200 specifically includes the following steps:

[0083] Step S201, performing Hough transform on the image to be processed to obtain at least one long horizontal straight line in the image to be processed, wherein the length of the long horizontal straight line is greater than the length threshold, and the length threshold is half of the width of the image to be processed.

[0084] Step S202: The longest one among all the above long horizontal straight lines is used as the above target straight line.

[0085] Step S203, performing tilt angle correction on the image to be processed according to the target straight line and the text information to obtain a corrected image, and performing tilt angle correction on the text information to obtain text correction information.

[0086] Specifically, for the original image to be processed, Hough transform is used to obtain the long straight line in the figure. The text information obtained during character recognition can be combined to obtain the angle of image inclination, so that the image to be processed is rotated to obtain the corrected image after correction. In the corrected image, relative to the pre-established world coordinate system, the table is from top to bottom in the vertical direction and from left to right in the horizontal direction.

[0087] The above-mentioned length threshold is a pre-set threshold for determining a straight line. The above-mentioned length threshold can be a pre-set fixed value, or can be pre-set to half the width of the image to be processed, and can also be adjusted according to actual needs, which is not specifically limited here. The above-mentioned horizontal long straight line is a straight line along the horizontal axis of the coordinate system, that is, it can be considered as a horizontal line in the table. Specifically, multiple horizontal long straight lines may be obtained, but in this embodiment, only the longest horizontal long straight line is retained as the target straight line to be used in the subsequent judgment process.

[0088] Further, in this embodiment, the above-mentioned step S203 includes: obtaining the first image tilt angle according to the length of the above-mentioned text content and the above-mentioned text box coordinates, and obtaining the second image tilt angle according to the above-mentioned target straight line; obtaining the target image tilt angle according to the above-mentioned first image tilt angle and the above-mentioned second image tilt angle, wherein the above-mentioned target image tilt angle is the maximum value of the first absolute value and the second absolute value, the above-mentioned first absolute value is the absolute value of the above-mentioned first image tilt angle, and the above-mentioned second absolute value is the absolute value of the above-mentioned second image tilt angle; rotating the above-mentioned image to be processed according to the above-mentioned target image tilt angle to obtain a corrected image; performing tilt angle correction on the text box coordinates in the above-mentioned text information according to the above-mentioned target image tilt angle to obtain text box correction coordinates, and using the text content corresponding to each of the above-mentioned text box correction coordinates as the text correction content in the above-mentioned text correction information to construct the above-mentioned text correction information.

[0089] The first image tilt angle is the tilt angle of the bottom of the text box in the preset world coordinate system (the angle formed with the positive direction of the horizontal axis). The text box is surrounded by four sides consisting of four coordinates. When performing OCR recognition, the up and down directions of each character in the text content can be determined, and the side below the character is used as the bottom of the text box.

[0090] Specifically, by combining the length of the text content with the position of each text box, it is possible to determine whether most of the text boxes in the image to be processed are horizontal or vertical, determine the image orientation, and calculate the first image tilt angle of the image to be processed based on the text box coordinates of this part of the text boxes.

[0091] In one application scenario, if multiple tilt angles are calculated based on multiple text contents and their corresponding text box coordinates, the average value of all tilt angles can be used as the tilt angle of the first image to improve accuracy, or the tilt angle with the highest proportion among all tilt angles can be used as the tilt angle of the first image. No specific limitation is made here.

[0092] In this embodiment, the second image tilt angle is the tilt angle corresponding to the target straight line, that is, the angle between the target straight line and the positive direction of the horizontal axis of the coordinate system.

[0093] After obtaining the above target tilt angle, the original image to be processed is rotated to obtain the corresponding corrected image. The corrected image still contains the previous text content and each text box, but with the rotation of the image, the coordinates of the text box have changed. At this time, the coordinates of each text box can be corrected according to the rotation angle to obtain the corresponding text box correction coordinates. At the same time, the text content in each text box is used as the text correction content, the corresponding relationship between the text correction content and the text correction coordinates is determined, and the text correction information is constructed.

[0094] In one application scenario, the corrected text content and text box coordinates can also be combined into a dictionary format for output to facilitate search. The above dictionary format includes multiple text correction information, and one text correction information includes the specific characters of the text content and the four coordinates of the corresponding text box. For example, a text correction information corresponding to the dictionary format is: {'text': 'text content', 'text_region': [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]}.

[0095] Step S300, respectively obtain each denoising table area in the above-mentioned corrected image according to the pre-trained table area detection model, crop and splice the above-mentioned corrected image according to the above-mentioned denoising table area to obtain the target denoised image, and obtain text denoising information according to the above-mentioned target denoised image and the above-mentioned text correction information.

[0096] The target denoised image only includes all the denoising table areas, and the text denoising information includes the text denoising content within the denoising table area and the text box denoising coordinates corresponding to the text denoising content.

[0097] It should be noted that in addition to the table, the above-mentioned image to be processed may also include a lot of other text outside the table. This part of the text does not need to be identified and extracted during the table recognition process, and this part of the text will also affect the efficiency and / or accuracy of the table recognition. Therefore, in this embodiment, the text outside the table and the content of the corresponding area in the corrected image can be deleted through the noise reduction process, thereby improving the efficiency and accuracy of table recognition. Specifically, the table area in the corrected image is identified, and the content outside the table area in the corrected image and the corrected text information are deleted, and only the relevant content within the table area is retained, so that the table position and information can be accurately obtained, which is convenient for subsequent table management.

[0098] Specifically, the above step S300 includes: obtaining each table area in the above-mentioned corrected image according to the above-mentioned pre-trained table area detection model; obtaining each table unit frame area in the above-mentioned corrected image according to the pre-trained table unit frame detection model; obtaining the above-mentioned denoising table area according to the above-mentioned table area and the above-mentioned table unit frame area; cropping and splicing the above-mentioned corrected image according to the above-mentioned denoising table area to obtain the target denoising image; using the above-mentioned text correction content belonging to the above-mentioned denoising table area in the above-mentioned text correction information as the above-mentioned text denoising content, and obtaining the text box denoising coordinates corresponding to each of the above-mentioned text denoising contents according to the above-mentioned target denoising image, the above-mentioned corrected image and the above-mentioned text box correction coordinates.

[0099] Among them, the above-mentioned pre-trained table area detection model and table cell frame detection model are pre-trained deep learning models based on the Mask RCNN target detection algorithm, which can be used to detect areas where tables may exist in images or corresponding table cell frames. The input of the above-mentioned table area detection model is an image (i.e., a corrected image), and the corresponding output is the table area in the image (i.e., the image position coordinates containing the table). The input of the above-mentioned table cell frame detection model is an image (i.e., a corrected image), and the corresponding output is each table cell frame area in the table (i.e., the position coordinates corresponding to each cell frame in the table).

[0100] It should be noted that in one application scenario, only the table area detection model can be used to detect and obtain the corresponding table area as the denoised table area. However, in this embodiment, the denoised table area is obtained by combining the table area and the table cell frame area, that is, the table area and the table cell frame area are merged to obtain the denoised table area, thereby improving the accuracy of table recognition.

[0101] Furthermore, an image including only the denoising table area and the corresponding character recognition result are obtained according to the denoising table area to complete the denoising. Specifically, the corrected image is cropped, and the image content corresponding to each denoising table area in the corrected image is cropped and spliced ​​to obtain the target denoised image. It should be noted that compared with the corrected image, the position of a part of the text information in the obtained target denoised image has changed. Therefore, the text box correction coordinates are transformed to obtain the corresponding text box denoising coordinates, and the text denoising content corresponding to each text box denoising coordinate can also be determined according to the corresponding relationship in the original text correction information, thereby realizing the construction of text denoising content.

[0102] It should be noted that the purpose of noise reduction is to obtain the precise table position and avoid information interference in non-table areas. After cropping, the tables can be spliced ​​in sequence according to their vertical (or horizontal) positions. After noise reduction, an image containing only the target table area and the text noise reduction information in this image is obtained.

[0103] Step S400: performing table structure recognition based on the text denoising information and the target denoising image to obtain a plurality of target tables.

[0104] Specifically, after obtaining the above-mentioned target denoised image and text denoised information, table structure recognition based on multimodal input can be performed to obtain the corresponding target table. Among them, the above-mentioned table structure recognition based on multimodal input refers to using a pre-trained table structure recognition model to perform table recognition, and the input of the above-mentioned table structure recognition model is multi-mode data, and the above-mentioned multi-mode data includes images, specific text content and the position corresponding to the text content (i.e., the corresponding text box coordinates). Therefore, in this embodiment, the above-mentioned table structure recognition model is a model based on multimodal input, that is, a model that can process multimodal input data.

[0105] Specifically, the step S400 includes: obtaining text denoising contents in each table unit frame area in the target denoised image, respectively performing row and column splicing adjustments on the text denoising contents in each table unit frame area to obtain each text splicing content, and obtaining text frame splicing coordinates corresponding to each text splicing content, and constructing text splicing information composed of the text splicing contents and the text frame splicing coordinates;

[0106] The target table is obtained according to the above text splicing information, the above target denoised image and the pre-trained table structure recognition model.

[0107] In this embodiment, the target table obtained above is a table in a format that can be edited and operated interactively or directly, such as an EXCEL format.

[0108] It should be noted that a table cell obtained in the above processing may include one text box or multiple text boxes (this embodiment is explained by taking one text box as an example). In the process of OCR recognition, the text content in a text box is not merged, that is, there may be text content with line breaks in each text box, and the text noise reduction content obtained above is also not merged. However, in the actual application process, the meaning of the text noise reduction content in a table cell box is continuous. Therefore, the splicing of character recognition results can be achieved based on the table cell box area. Specifically, based on each table cell box, the scattered rows and columns in the text noise reduction content in each table cell box area are spliced ​​to restore its semantics in a table cell box, which is conducive to the subsequent correction of the text. The position of the spliced ​​text splicing content can still be determined according to the coordinates of the corresponding text box, so that the corresponding text splicing information can be constructed.

[0109] The step of obtaining the target table according to the text splicing information, the target denoised image and a pre-trained table structure recognition model includes:

[0110] The target table is obtained by processing through preset loop processing steps, wherein the i-th loop processing step includes:

[0111] Inputting the input data into the table structure recognition model to perform table recognition and obtain the to-be-processed information of the recognized reconstructed table, wherein the to-be-processed information includes the cell row and column information, cell coordinates and cell text content of the reconstructed table;

[0112] Correct the text content of each cell in the reconstructed table according to the preset knowledge base to obtain a first corrected table;

[0113] Acquire the form confirmation information input by the target object, and when the form confirmation information indicates successful recognition, use the first revised form as the target form, output the target form and the loop ends;

[0114] When the table confirmation information indicates that the recognition is incorrect, the table modification instruction data input by the target object is obtained, the first modified table is modified according to the table modification instruction data to obtain a second modified table, and a modified image, cell text content and cell coordinates corresponding to the second modified table are obtained, and the modified image, cell text content and cell coordinates corresponding to the second modified table are used as input data of the table structure recognition model and the i+1th cycle is performed;

[0115] Wherein, i is an integer greater than or equal to 1, and the input data of the above table structure recognition model in the first cycle includes the above text splicing information and the above target denoised image.

[0116] Among them, the table structure recognition model is a pre-trained multi-modal input deep learning model, whose input data includes data of multiple models, such as images and text information. Correspondingly, the table structure recognition model is pre-trained and can be used to recognize the table in the input image according to the input data, obtain the reconstructed table and the cell row and column information, cell coordinates and cell text content of the reconstructed table, and specifically, the table structure recognition model can encode and convert the input text position and image information to obtain whether the texts in the image are connected in rows or columns. The target object is a user or device that needs to judge and correct the table.

[0117] Specifically, the input data (such as the above-mentioned text splicing information and the above-mentioned target denoised image) is input into the table structure recognition model to obtain a reconstructed table, and then the reconstructed table is converted to obtain the structural information and content information of the reconstructed table. The structural and content information are the cell row and column information, cell coordinates and cell text content of the corresponding reconstructed table.

[0118] In one application scenario, the formula for calculating the row-column relationship is: R row ,R col =f(t,p,b). Where f represents the pre-trained deep learning table structure recognition model, t is the list of corresponding text content in the input data, p is the image containing the table in the input data, b is the list of corresponding text box coordinates in the input data, and R row Represents the output table row relationship, R col Represents the relationship between the output table columns. The row-column relationship is the row-column relationship between text and adjacent text. If they are in the same row, they are row relationships; if they are in the same column, they are column relationships; if they are not in the same row or column, they are irrelevant.

[0119] Specifically, based on the predicted row and column relationships between the texts, the texts are calculated row by row / column by column to obtain a table structure. The structure and content information of the table includes the row and column corresponding to the cell, the coordinates of the cell and the text content in the cell. In this way, the structure and content information of the table obtained relative to the input data obtains the specific row and column positions of the table, rather than just the row and column relationships between each text.

[0120] It should be noted that in this embodiment, the form will be repeatedly corrected according to the form confirmation information received from the target object to improve the accuracy of form recognition. The correction process includes correction based on the preset knowledge base and correction according to the form modification instruction data received from the target object.

[0121] In an application scenario, when the form confirmation information input by the target object is not obtained, the loop may be directly terminated and the corresponding first revised form may be output.

[0122] Specifically, the above-mentioned correcting the text content of each cell in the above-mentioned reconstructed table according to the preset knowledge base to obtain the first corrected table includes:

[0123] Obtain a preset knowledge base, preset tabulation rules, and pre-trained language models;

[0124] Extract keywords from the above knowledge base, and obtain row and column association information corresponding to each of the above keywords according to the above tabulation rules;

[0125] Correcting the text content of each cell of the reconstructed table according to the keywords and the row-column association information to obtain a first corrected text;

[0126] Acquire a first word confidence corresponding to the first revised text according to the input data and the first revised text, and revise the first revised text according to the first word confidence and the language model to obtain a second revised text;

[0127] Obtaining the second word confidence corresponding to the second revised text according to the input data, performing glyph matching and correction on the second revised text according to the second word confidence, the keyword and a preset glyph similarity threshold, to obtain a third revised text;

[0128] The first correction table is generated according to the cell row and column information, the cell coordinates and the third correction text.

[0129] The preset knowledge base is a knowledge base preset by the user, and the content in the knowledge base includes common information of the corresponding field preset by the user. For example, the corresponding field may be the financial field, and the corresponding common information may be company names, bond names, etc. in the financial field. The preset tabulation rules are common tabulation rules of the corresponding field preset by the user, such as which information can usually be used as row information, which can be used as column information, which information has a corresponding relationship, etc. The pre-trained language model is a pre-trained BERT model, which can be used to determine the possible error position according to the input text and give the value corresponding to the possible error position.

[0130] In this embodiment, an index is first established based on the knowledge base content. Specifically, the knowledge base content is split and modeled using methods such as the Chinese word segmentation model, and keywords in the knowledge base are extracted, so that the database index can be optimized, the search in the subsequent matching process can be reduced, and the table recognition efficiency can be improved. Then, the knowledge is associated with the table row and column structure, and the extracted keywords are associated with the row and column information according to the tabulation rules.

[0131] Furthermore, the content of the reconstructed table is fuzzy matched using the knowledge base content (i.e., keywords), and the content of the reconstructed table is corrected to obtain the first corrected text. Specifically, the table content is corrected based on the association between keywords and table rows and columns, so as to achieve better table recognition effects for the corresponding fields. For example, if "Today's Sun is Big Company" appears in the text, and its corresponding row / column title is the field corresponding to the knowledge base, this information does not exist in the knowledge base, and only "Today's Sun is Big Company" exists. It is judged that its similarity is 87.5%, which is greater than the set threshold, and it is corrected to "Today's Sun is Big Company".

[0132] Specifically, the first revised text can be further revised according to the language model to obtain the second revised text. The table structure and content in the input data and the first single word confidence corresponding to the table (which can be obtained by performing OCR recognition on the image in the input data) are input into the language model, and the content in the first revised text is revised according to the confidence output by the language model and the preset confidence threshold. For example, when the first revised text is ['the sun in the sky'], and the corresponding single word confidence is [0.99, 0.99, 0.99, 0.02, 0.98], the confidence of the word 'big' is lower than 0.8 (the preset minimum single word confidence threshold), and it is judged that it is wrong, and the language model is used to predict the possible value of the position, wherein the confidence of "the sun in the sky" is the highest, which is 0.70, which is greater than the preset confidence threshold (such as 0.6), and the result is used to replace "the sun in the sky".

[0133] Further, the above-mentioned second corrected text can also be corrected and revised according to keywords to obtain the corresponding third corrected text. Specifically, a second single-character confidence level is obtained based on the input data and the second corrected text. The content of the table is further corrected according to the keywords and the second single-character confidence level. According to the glyph similarity calculation method, the content (the content corresponding to the second corrected text) is corrected and revised to obtain the third corrected text. Determine the second single-character confidence level to determine whether the second corrected text is incorrect. If it is incorrect, match it with the keywords, and calculate the similarity between the incorrect character part and the corresponding position in the thesaurus. If the similarity is greater than the set similarity threshold, replace it. For example, if the single-character confidence level in the text is lower than 0.8 (the preset minimum single-character confidence level), it is determined that there is an error in this position of the text. If there is "Tiananmen Square in Beijing" in the thesaurus, the corresponding second corrected text is ["Beijing Great Tiananmen Square"], and the second single-character confidence levels are [0.99, 0.99, 0.02, 0.99, 0.99]. Match "Tiananmen Square in Beijing" in the keyword library. The character similarity between "Tian" and "Great" is 0.75, which is greater than 0.6 (the preset similarity threshold). It is determined that this character is incorrect and the word in the thesaurus can replace this incorrect text, so as to make a correction.

[0134] It should be noted that the steps of obtaining the preset knowledge base, the preset tabulation rules, the pre-trained language model, and extracting the keywords in the above knowledge base, and obtaining the row-column association information corresponding to each of the above keywords according to the above tabulation rules can be executed only once and the results can be saved, and can be directly called in the subsequent process. In this way, the required time can be reduced and the table recognition efficiency can be improved. The above-mentioned single-character confidence levels are the results given after the corresponding images are recognized by OCR, that is, the possibility that this area can be a certain character. If the possibility is too low, the character may be misrecognized. The confidence level output by the language model is the similarity between the character supplemented at the possible error position and the original character in terms of glyph.

[0135] In this embodiment, an interactive interface can also be provided for the user, and the user can adjust the target table and the first corrected table on the interactive interface. Specifically, when adjusting the first corrected table, the user inputs table confirmation information and table modification instruction data through the interactive interface, and obtains the second corrected table after modification.

[0136] In an application scenario, the recognized table position and the content of the corresponding first corrected table are marked on the original image to be processed. The content of the first corrected table includes movable and adjustable text boxes and editable table content presented in the form of EXCEL. The table modification instruction data input by the user can include deletion instruction data, new addition instruction data, correction instruction data, cell movement specification data, cell merging instruction data, etc.

[0137] For example, the user can select unnecessary information with one click and delete it. For example, the user clicks the left mouse button to select the unnecessary table cells in the first correction table, releases the left button to successfully select, and then deletes the selected cells with one click. The user can also add the information that needs to be added or modify the erroneous text content. Specifically, according to the user's needs, new cells can be drawn on the image to be processed, cells can be moved and merged, and the erroneous cell information after identification can be modified to modify its text content.

[0138] Figure 3 is a schematic diagram of selecting and deleting a table through an interactive interface provided by an embodiment of the present invention, such as Figure 3 As shown, for a table (first revised table) identified and displayed on the interactive interface, the user can select some cells therein and delete the selected parts, and then input the processed table into the table structure recognition model for re-recognition to re-recognize the table structure. Figure 3 In the text field, Text represents the specific text content, col1, col2, and col3 represent the first, second, and third columns respectively, and row1, row2, and row3 represent the first, second, and third rows respectively.

[0139] A second revised form can be obtained through user correction. In order to further improve the accuracy of form recognition, the second revised form can be used as input data of the form recognition structure model and corrected again. This cycle continues until the form recognition is confirmed to be correct based on the form confirmation information.

[0140] A corrected image in the area of ​​the second corrected table is obtained based on the denoised image, and the cell information in the second corrected table and the corrected image are input into a table structure recognition model based on multimodal input to reconstruct the row and column structure of the second corrected table, and the above cycle is repeated until the cycle ends. It should be noted that the corresponding text content in the cell information in the second corrected table is the content after the above correction steps, so it does not need to be obtained through OCR recognition.

[0141] For the target table finally obtained, the table structure and content are displayed in the interactive interface in the form of a table, and when the user selects one or several cells, the corresponding table cell frame will light up, which is convenient for the user to observe and use.

[0142] In this embodiment, the above table recognition method is further specifically described based on a specific application scenario. Figure 4 is a specific flow chart of a table recognition method provided by an embodiment of the present invention, such as Figure 4As shown, in this embodiment, character recognition is first performed on an image containing a table (i.e., an image to be processed) to obtain text information corresponding to the image, and then the character recognition result and the image feature are fused to perform image correction, and further image noise reduction is performed based on table detection. Then, table structure recognition based on multimodal input is performed, and then table content correction based on the knowledge base is performed. It is determined whether the table recognition is successful based on the table confirmation information. If it is unsuccessful, the table modification instruction data input by the user is obtained and the table is adjusted, and then the table structure recognition based on multimodal input is returned again, and the cycle is repeated. The cycle stops until the table recognition is judged to be successful based on the table confirmation information, and the successfully recognized table is output.

[0143] As can be seen from the above, in the table recognition method provided by the embodiment of the present invention, an image to be processed is obtained, character recognition is performed on the image to be processed, and text information in the image to be processed is obtained, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates; image features of the image to be processed are obtained, and according to the image features and the text information, a tilt angle correction is performed on the image to be processed to obtain a corrected image, and a tilt angle correction is performed on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, the length of the target straight line is greater than a preset length threshold, and the text correction information The information includes text correction content and text box correction coordinates; each denoising table area in the above-mentioned corrected image is obtained respectively according to a pre-trained table area detection model, the above-mentioned corrected image is cropped and spliced ​​according to the above-mentioned denoising table area to obtain a target denoising image, and text denoising information is obtained according to the above-mentioned target denoising image and the above-mentioned text correction information, wherein the above-mentioned target denoising image only includes all the above-mentioned denoising table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content; table structure recognition is performed according to the above-mentioned text denoising information and the above-mentioned target denoising image to obtain multiple target tables.

[0144] Compared with the prior art which requires manual recognition of tables in images and relies on manual production of corresponding tables on electronic devices, the present invention can automatically recognize tables in images to be processed, obtain text information in the image to be processed, perform tilt angle correction and noise reduction on the image to be processed and the text information obtained by recognition, and finally obtain the target table in the image to be processed, without the need for manual recognition of the contents of the image one by one, which is conducive to improving the efficiency of table recognition.

[0145] Furthermore, by adopting the method in this embodiment, after obtaining the knowledge base corresponding to the production environment, the knowledge base can be associated with the table information, and the recognized information can be corrected using multimodal input, thereby improving the performance of the method in table scenarios in various fields, and being able to adapt to the table recognition requirements in different production environments in different fields, thereby improving the effect of table recognition. The time loss caused by manual table recognition is reduced, and an interactive page is provided to facilitate the operator to select the required table area to obtain the table structure, thereby improving production speed.

[0146] Exemplary Devices

[0147] like Figure 5 As shown in , corresponding to the above table recognition method, an embodiment of the present invention further provides a table recognition system, and the above table recognition system includes:

[0148] The image processing module 510 is used to obtain the image to be processed, perform character recognition on the image to be processed, and obtain text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates.

[0149] The correction module 520 is used to obtain the image features of the above-mentioned image to be processed, and perform tilt angle correction on the above-mentioned image to be processed to obtain a corrected image based on the above-mentioned image features and the above-mentioned text information, and perform tilt angle correction on the above-mentioned text information to obtain text correction information, wherein the above-mentioned image features include the target straight line in the above-mentioned image to be processed, and the length of the above-mentioned target straight line is greater than a preset length threshold, and the above-mentioned text correction information includes text correction content and text box correction coordinates.

[0150] The denoising module 530 is used to obtain each denoising table area in the above-mentioned corrected image according to a pre-trained table area detection model, crop and splice the above-mentioned corrected image according to the above-mentioned denoising table area to obtain a target denoised image, and obtain text denoising information according to the above-mentioned target denoising image and the above-mentioned text correction information, wherein the above-mentioned target denoising image only includes all the above-mentioned denoising table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content.

[0151] The table recognition module 540 is used to perform table structure recognition according to the text denoising information and the target denoising image and obtain a plurality of target tables.

[0152] It should be noted that the above system may also include an interactive module for realizing interaction with the user. For example, the user may input form confirmation information and form modification instruction data through the interactive module, or perform interactive behaviors such as selecting and editing a form. The above interactive module may also be provided with a display interface for displaying the target form.

[0153] Specifically, in the table recognition system, the image to be processed is obtained, and operations such as character recognition, tilt angle correction, denoising, and table content correction are performed on it, and finally the target table is obtained and displayed. The displayed target table is visual and can be interacted and operated, and its table format is in the form of EXCEL. When performing tilt angle correction, the image is corrected to a tilt angle of 0. Optionally, the above interaction module uses the optimized labelme tool.

[0154] Specifically, in this embodiment, the specific functions of the above table recognition system and its modules can refer to the corresponding description in the above table recognition method, which will not be repeated here.

[0155] It should be noted that the division method of the various modules of the above table recognition system is not unique and is not specifically limited here.

[0156] Figure 6 is a schematic diagram of the structure of another table recognition system provided by an embodiment of the present invention. Figure 6As shown, in an application scenario, the system can be divided into an interactive module and a table recognition module, the interactive module includes an input unit, a knowledge base construction unit and an interactive unit, and the table recognition module includes a data preprocessing unit, a text recognition unit, a tilt correction and noise reduction unit, a table frame detection unit, a knowledge base fuzzy matching unit, a structure reconstruction unit and a data post-processing unit. During the processing, the input unit obtains an image containing a table input by the user, the data preprocessing unit is used to enhance the image, and the text recognition unit is used to perform OCR operation on the enhanced image to identify the text content and text box coordinates therein. The tilt correction and noise reduction unit is used to perform tilt angle correction and denoising on the enhanced image, and the table frame detection unit detects and locates the table frame according to the target denoised image, text denoising content and text box denoising coordinates output by the denoising unit. The user can also input the collected industry knowledge into the system in advance and build the corresponding knowledge base through the knowledge base construction unit. The knowledge base fuzzy matching unit is used to perform fuzzy matching and error correction on the text information in the table according to the constructed knowledge base, the text denoising content obtained by recognition and the input of the interactive unit. The structure reconstruction unit reconstructs the table structure according to the text box position and text content output by the knowledge base fuzzy matching unit and the image information of the table area on the image, and outputs the reconstructed table. The data post-processing unit outputs the data format required by the interactive system according to the table output by the structure reconstruction unit, and outputs it to the interactive unit. The interactive unit displays the table and can be operated by the user.

[0157] Furthermore, the interactive module inputs the cropped table image and the processed cell position information into the display interface of the interactive unit to obtain a visual and further operable annotation result. After the user performs further operations on the result on the display interface, the result and the knowledge of the knowledge base construction module can be transferred back to the table recognition module to reconstruct the table structure again, and a new table structure and content recognition result can be obtained, which is displayed in the form of an EXCEL table on the display interface.

[0158] The table recognition module is used to recognize the input table. Specifically, the table image can be corrected to correct the image to a tilt angle of 0; the character recognition operation is performed on the input original image to identify the text content and text position information in the original image; the table position and the table frame position of the wired table are detected to further locate the table position in the future; the table area detection and cropping of the corrected image are performed to crop the area containing the table in the image to reduce noise in the future; the corrected text content and position and the table area image are used as input, and the table structure is reconstructed using a table structure reconstruction model based on multimodal input, and the reconstructed cells are output. According to the domain knowledge and associated information input by the operator, the fuzzy matching method is used to correct the table content and output the corrected table; the corrected table is processed into the data format required in the interactive module. The data format required in the interactive module is the shape object defined in the PYQT for building the front end. Its coordinates, content, color and other information need to be defined, and the text and position information in the corresponding table are extracted and assigned to the corresponding shape object.

[0159] In this way, based on the table recognition system provided by this embodiment, the knowledge base can be associated with the table information, and the recognized information can be corrected using multimodal input, thereby improving the performance in the table recognition scenarios in various corresponding fields, adapting to the table requirements in different production environments in different fields, and improving the table recognition effect. It can also reduce the time loss caused by manual table recognition, and provide an interactive interface to facilitate user operation and adjustment, thereby improving user experience and work efficiency.

[0160] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a table recognition program. The internal memory provides an environment for the operation of the operating system and the table recognition program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the table recognition program is executed by the processor, the steps of any one of the above-mentioned table recognition methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0161] Those skilled in the art will understand that Figure 7The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a smart terminal is provided. The smart terminal includes a memory, a processor, and a table recognition program stored in the memory and executable on the processor. When the table recognition program is executed by the processor, the following operation instructions are performed:

[0163] Acquire an image to be processed, perform character recognition on the image to be processed, and acquire text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates;

[0164] Acquire the image features of the image to be processed, and perform tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and perform tilt angle correction on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, and the length of the target straight line is greater than a preset length threshold, and the text correction information includes text correction content and text box correction coordinates;

[0165] According to the pre-trained table area detection model, each denoised table area in the above-mentioned corrected image is respectively obtained, and according to the above-mentioned denoised table area, the above-mentioned corrected image is cropped and spliced ​​to obtain a target denoised image, and the text denoising information is obtained according to the above-mentioned target denoised image and the above-mentioned text correction information, wherein the above-mentioned target denoised image only includes all the above-mentioned denoised table areas, and the above-mentioned text denoising information includes the text denoising content within the range of the above-mentioned denoising table area and the text box denoising coordinates corresponding to the above-mentioned text denoising content;

[0166] Table structure recognition is performed based on the text denoising information and the target denoising image to obtain multiple target tables.

[0167] An embodiment of the present invention further provides a computer-readable storage medium, on which a table recognition program is stored. When the table recognition program is executed by a processor, the steps of any table recognition method provided by the embodiment of the present invention are implemented.

[0168] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0169] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] In the embodiments provided by the present invention, it should be understood that the disclosed system / terminal device and method can be implemented in other ways. For example, the system / terminal device embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0173] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0174] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A table recognition method, characterized in that: The method comprises: Acquire an image to be processed, perform character recognition on the image to be processed, and acquire text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates; Acquire image features of the image to be processed, and perform tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and perform tilt angle correction on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, and the length of the target straight line is greater than a preset length threshold, and the text correction information includes text correction content and text box correction coordinates; According to a pre-trained table area detection model, each denoising table area in the corrected image is respectively obtained, and according to the denoising table area, the corrected image is cropped and spliced ​​to obtain a target denoised image, and text denoising information is obtained according to the target denoised image and the text correction information, wherein the target denoised image only includes all the denoising table areas, and the text denoising information includes text denoising content within the denoising table area and text box denoising coordinates corresponding to the text denoising content; Table structure recognition is performed according to the text denoising information and the target denoising image to obtain a plurality of target tables.

2. The table recognition method according to claim 1, characterized in that: The step of acquiring the image features of the image to be processed, and performing tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and performing tilt angle correction on the text information to obtain text correction information includes: Performing Hough transform on the image to be processed to obtain at least one long horizontal straight line in the image to be processed, wherein the length of the long horizontal straight line is greater than the length threshold, and the length threshold is half of the width of the image to be processed; The longest of all the horizontal long straight lines is used as the target straight line; The image to be processed is subjected to tilt angle correction according to the target straight line and the text information to obtain a corrected image, and the text information is subjected to tilt angle correction to obtain text correction information.

3. The table recognition method according to claim 2, characterized in that: The step of performing tilt angle correction on the image to be processed according to the target straight line and the text information to obtain a corrected image, and performing tilt angle correction on the text information to obtain text correction information, comprises: The first image tilt angle is calculated based on the length of the text content and the text box coordinates, and the second image tilt angle is calculated based on the target straight line; Acquire a target image tilt angle according to the first image tilt angle and the second image tilt angle, wherein the target image tilt angle is a maximum value of a first absolute value and a second absolute value, the first absolute value is an absolute value of the first image tilt angle, and the second absolute value is an absolute value of the second image tilt angle; Rotating the image to be processed according to the tilt angle of the target image to obtain a corrected image; The text box coordinates in the text information are corrected for inclination angle according to the inclination angle of the target image to obtain text box correction coordinates, and the text content corresponding to each of the text box correction coordinates is used as the text correction content in the text correction information to construct the text correction information.

4. The table recognition method according to claim 1, characterized in that: The method of obtaining the denoising table areas in the corrected image according to the pre-trained table area detection model, cropping and splicing the corrected image according to the denoising table areas to obtain a target denoised image, and obtaining text denoising information according to the target denoised image and the text correction information includes: Acquire each table area in the corrected image according to the pre-trained table area detection model; Acquire each table cell frame area in the corrected image according to a pre-trained table cell frame detection model; Acquire the noise reduction table area according to the table area and the table cell frame area; Cropping and splicing the corrected image according to the denoising table area to obtain a target denoised image; The text correction content in the text correction information that belongs to the denoising table area is used as the text denoising content, and text box denoising coordinates corresponding to each text denoising content are obtained according to the target denoising image, the corrected image and the text box correction coordinates.

5. The table recognition method according to claim 4, characterized in that: The step of performing table structure recognition according to the text denoising information and the target denoising image and obtaining a plurality of target tables includes: Acquire the text denoising content in each table unit frame area in the target denoised image, and respectively perform row and column splicing adjustments on the text denoising content in each table unit frame area to obtain each text splicing content, obtain the text frame splicing coordinates corresponding to each text splicing content, and construct text splicing information consisting of the text splicing content and the text frame splicing coordinates; A target table is obtained according to the text splicing information, the target denoised image and a pre-trained table structure recognition model.

6. The table recognition method according to claim 5, characterized in that: The step of obtaining a target table according to the text splicing information, the target denoised image and a pre-trained table structure recognition model includes: The target table is obtained by processing through preset loop processing steps, wherein the i-th loop processing step includes: Inputting input data into the table structure recognition model to perform table recognition and obtain the to-be-processed information of the recognized reconstructed table, wherein the to-be-processed information includes the cell row and column information, cell coordinates, and cell text content of the reconstructed table; Correcting the text content of each cell in the reconstructed table according to a preset knowledge base to obtain a first corrected table; Acquire the form confirmation information input by the target object, and when the form confirmation information indicates successful recognition, use the first revised form as the target form, output the target form and the loop ends; When the table confirmation information indicates that the recognition is incorrect, the table modification instruction data input by the target object is obtained, the first modified table is modified according to the table modification instruction data to obtain a second modified table, and a modified image, cell text content and cell coordinates corresponding to the second modified table are obtained, and the modified image, cell text content and cell coordinates corresponding to the second modified table are used as input data of the table structure recognition model and the i+1th cycle is performed; Wherein, i is an integer greater than or equal to 1, and in the first cycle, the input data of the table structure recognition model includes the text splicing information and the target denoised image.

7. The table recognition method according to claim 6, characterized in that: The step of correcting the text content of each cell in the reconstructed table according to the preset knowledge base to obtain a first corrected table includes: Obtain a preset knowledge base, preset tabulation rules, and pre-trained language models; Extracting keywords from the knowledge base, and obtaining row and column association information corresponding to each keyword according to the tabulation rule; Correcting the text content of each cell of the reconstructed table according to the keyword and the row-column association information to obtain a first corrected text; Acquire a first word confidence corresponding to the first revised text according to the input data and the first revised text, and revise the first revised text according to the first word confidence and the language model to obtain a second revised text; Acquire a second single-word confidence corresponding to the second revised text according to the input data, and perform glyph matching and correction on the second revised text according to the second single-word confidence, the keyword, and a preset glyph similarity threshold to obtain a third revised text; The first correction table is generated according to the cell row and column information, the cell coordinates and the third correction text.

8. A table recognition system, characterized in that: The system comprises: The image processing module is used to obtain the image to be processed, perform character recognition on the image to be processed, and obtain text information in the image to be processed, wherein the image to be processed includes at least one table, and the text information includes text content and text box coordinates; A correction module, used for acquiring image features of the image to be processed, and performing tilt angle correction on the image to be processed to obtain a corrected image according to the image features and the text information, and performing tilt angle correction on the text information to obtain text correction information, wherein the image features include a target straight line in the image to be processed, the length of the target straight line is greater than a preset length threshold, and the text correction information includes text correction content and text box correction coordinates; A denoising module, for respectively acquiring each denoising table area in the corrected image according to a pre-trained table area detection model, cropping and splicing the corrected image according to the denoising table area to obtain a target denoised image, and obtaining text denoising information according to the target denoised image and the text correction information, wherein the target denoised image only includes all the denoising table areas, and the text denoising information includes text denoising content within the denoising table area and text box denoising coordinates corresponding to the text denoising content; The table recognition module is used to perform table structure recognition according to the text denoising information and the target denoising image and obtain a plurality of target tables.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a table recognition program stored in the memory and executable on the processor. When the table recognition program is executed by the processor, the steps of the table recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a table recognition program, and when the table recognition program is executed by the processor, the steps of the table recognition method according to any one of claims 1 to 7 are implemented.

Images