Drawing table recognition system and method and computer program product

Abstract

The invention discloses a drawing table recognition system, which comprises a drawing coordinate acquisition module used for inputting a drawing into a table area detection model to obtain vertex coordinates of each cell in the drawing, and calculating center point coordinates of each cell according to the vertex coordinates of each cell; the text area detection module is used for inputting the drawing into the OCR model to obtain vertex coordinates and text content of each textbox in the drawing, and calculating center point coordinates of each textbox according to the vertex coordinates of each textbox; and the textbox and cell matching module is used for sequentially calculating the distance between the central point coordinate of each textbox and the central point coordinate of each cell, and matching the textbox with the minimum distance between the central point coordinate of the textbox and the central point coordinate of the cell with the cell. According to the method, the table region detection model and the OCR model are combined, and the matching relation between the textbox and the cells is accurately restored.

Description

Technical Field

[0001] The present invention relates to the technical field of drawing recognition, and in particular to a drawing table recognition system, method and computer program product. Background Art

[0002] There are many challenges in the current project management of the construction industry, one of which is the problem of transcribing the engineering quantities and material quantities in the drawing tables. This link requires a lot of manual operations, which is time-consuming and prone to errors, causing great trouble to the project department engineers.

[0003] The traditional offline manual transcription method can no longer meet the current project management requirements for drawing transcription speed and quality. There is also the problem of the coexistence of tables and text boxes. A single OCR recognition tool cannot meet the needs of accurate recognition and cannot match text content with tables. Summary of the invention

[0004] The purpose of the present invention is to provide a drawing table recognition system to address the defects of the prior art, including:

[0005] The drawing coordinate acquisition module is used to input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell;

[0006] The text area detection module is used to input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box;

[0007] The text box and cell matching module is used to calculate the distance between the center point coordinates of each text box and the center point coordinates of each cell in turn, match the text box and cell with the smallest distance between the center point coordinates of the text box and the center point coordinates of the cell, and finally fill the text content in the text box into the cell that matches the text box.

[0008] Furthermore, it also includes: a table structure analysis module, which is used to sort each cell vertically and horizontally according to the vertex coordinates of each cell, obtain the row number of each cell, and calculate the number of columns of all rows, using the maximum column number of the row containing the maximum number of columns as the maximum column number for column number naming, and determining the column number of the corresponding cell according to the vertex coordinates of each cell.

[0009] Furthermore, the specific method for calculating the maximum number of columns of all rows is as follows: calculate the number of cells in each row, the number of cells in each row is the number of columns in each row, compare the number of cells in all rows, and obtain the number of columns in all rows.

[0010] Furthermore, the table area detection model is a Cycle-CenterNet model.

[0011] Furthermore, the OCR model is a CRNN model.

[0012] Furthermore, in the drawing coordinate acquisition module, the specific method for calculating the center point coordinates of each cell according to the vertex coordinates of each cell is:

[0013] After inputting the drawing into the table area detection model, the coordinates of the four vertices of the table area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x1, y1) (x2, y2) (x3, y3) (x4, y4). The coordinates of the center point of the table area are obtained based on the coordinates of the four vertices of the table area.

[0014] In the text area detection module, the specific method for calculating the center point coordinates of each text box according to the vertex coordinates of each text box is:

[0015] After the drawing is input into the OCR model, the coordinates of the four vertices of the text box area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x ′ 1,y1 ′ )(x ′ 2,y2 ′ )(x ′ 3,y3 ′ )(x ′ 4,y4 ′ ), and get the center point coordinates based on the coordinates of the four vertices of the text box area.

[0016] Furthermore, the text area detection module is also used to train the OCR model using special characters before the drawing is input into the OCR model. The specific method is as follows:

[0017] Special symbols in various drawings are collected, and ttf font files are drawn based on the special symbols. The ttf font files are font files of the drawing software used to make the drawings to which the special symbols belong. Binary data is generated according to the ttf font files, and drawings are generated based on the binary data. The drawings are expanded and eroded using the OpenCV tool library, and all the expanded and eroded drawings are combined to obtain a training data set. The OCR text recognition model is trained based on the training data set.

[0018] A drawing table recognition method, comprising:

[0019] Input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell;

[0020] Input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box;

[0021] Calculate the distance between the center point coordinates of each text box and the center point coordinates of each cell in turn, match the text box and cell with the smallest distance between the center point coordinates of the text box and the cell, and finally fill the text content in the text box into the cell that matches the text box.

[0022] Furthermore, it also includes: sorting each cell vertically and horizontally according to the vertex coordinates of each cell to obtain the row number of each cell, and calculating the number of columns of all rows, using the maximum column number of the row containing the maximum number of columns as the maximum column number for column number naming, and determining the column number of the corresponding cell according to the vertex coordinates of each cell.

[0023] A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the above-mentioned drawing table recognition method is implemented.

[0024] The beneficial effects of the present invention are:

[0025] 1. The present invention combines the table area detection model and the OCR model to identify the positions of cells and text boxes respectively. By calculating the position of the center point of the table and the position of the center point of the cell, the matching relationship between the text box and the cell can be accurately restored. Therefore, not only can the table be identified separately, but also the text box can be identified, and the two can be matched through the position relationship to restore the content of each cell in the drawing.

[0026] 2. By naming the row and column numbers of cells according to certain rules, we can finally get the row and column numbers of each cell, which facilitates the reading of the table at work.

[0027] 3. Before recognition, the OCR model is trained with special characters to improve the recognition accuracy.

[0028] 4. The table area detection model uses the Cycle-CenterNet model, and the OCR model uses the CRNN model. The above two models can still accurately identify a large number of distorted drawings and tables in actual work and have excellent robustness, which is very important in actual work. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0030] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] Example 1

[0032] A drawing table recognition system, comprising:

[0033] The drawing coordinate acquisition module is used to input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell;

[0034] The text area detection module is used to input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box;

[0035] The text box and cell matching module is used to calculate the distance between the center point coordinates of each text box and the center point coordinates of each cell in turn, match the text box and cell with the smallest distance between the center point coordinates of the text box and the center point coordinates of the cell, and finally fill the text content in the text box into the cell that matches the text box.

[0036] The table structure analysis module is used to sort each cell vertically and horizontally according to the vertex coordinates of each cell, obtain the row number of each cell, and calculate the number of columns of all rows, and use the maximum column number of the row containing the maximum number of columns as the maximum column number for column naming, and determine the column number of the corresponding cell according to the vertex coordinates of each cell. If the maximum column number of the row containing the maximum number of columns is not used as the maximum column number for column naming, then if a table is edited and only 8 columns are edited, but the second row must have 10 columns to fit, then when the second row is put into the table, the 9th and 10th columns will be squeezed into the 8th column of the table.

[0037] (1) The specific method for calculating the number of columns of all rows is as follows:

[0038] Calculate the number of cells in each row, which is equal to the number of columns in each row. By arranging the cells vertically according to the coordinates, the vertical and horizontal positional relationship of the cells in the drawing is restored, so the number of rows and columns in the table can be obtained.

[0039] (2) The table area detection model is the Cycle-CenterNet model. This model solves the problems of inaccurate geometric prediction of drawings with extreme physical distortion and the defects in extracting the logical structure of misaligned tables. The Cycle-CenterNet method has significant advantages over other methods in table structure parsing. Using the TEDS metric evaluation, the accuracy of Cycle-CenterNet has been improved by 24.6%.

[0040] The OCR model is a CRNN model. The CRNN model supports the recognition of indefinite length text, has the ability to recognize words and scene text, has strong generalization ability, can be well generalized to new and unseen images, can learn end-to-end without manual feature extraction or other preprocessing steps, is robust to image distortion, and improves the practicality of the model in actual work applications.

[0041] The above Cycle-CenterNet model and CRNN model can still accurately recognize a large number of distorted drawings and tables in actual work, and have excellent robustness, which is very important in actual work.

[0042] (3) In the drawing coordinate acquisition module, the specific method for calculating the center point coordinates of each cell according to the vertex coordinates of each cell is:

[0043] After inputting the drawing into the table area detection model, the coordinates of the four vertices of the table area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x1, y1) (x2, y2) (x3, y3) (x4, y4). The coordinates of the center point of the table area are obtained based on the coordinates of the four vertices of the table area.

[0044] In the text area detection module, the specific method for calculating the center point coordinates of each text box according to the vertex coordinates of each text box is:

[0045] After the drawing is input into the OCR model, the coordinates of the four vertices of the text box area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x ′ 1,y1 ′ )(x ′ 2,y2 ′ )(x ′ 3,y3 ′ )(x ′ 4,y4 ′ ), and get the center point coordinates based on the coordinates of the four vertices of the text box area.

[0046] In actual work, tables and text boxes in drawings are generally rectangular, so you only need to get the four vertices of the rectangle to get the coordinates of the center point by calculation.

[0047] (4) The text region detection module is also used to train the OCR model using special characters before the drawing is input into the OCR model. The specific method is as follows:

[0048] Special symbols in various drawings are collected, and ttf font files are drawn based on the special symbols. The ttf font files are font files of the drawing software used to make the drawings to which the special symbols belong. Binary data is generated according to the ttf font files, and drawings are generated based on the binary data. The drawings are expanded and eroded using the OpenCV tool library, and all the expanded and eroded drawings are combined to obtain a training data set. The OCR text recognition model is trained based on the training data set.

[0049] The software used in the design of each drawing is different, and the font files corresponding to the steel bar characters and other special characters used by each software are also different. The same character may be fat, thin, tall, or short, so corresponding ttf font files will be produced based on different drawing software to improve versatility.

[0050] In the construction industry, there are a large number of special characters. By using the OpenCV tool library to expand and erode the drawings, the skeletons of various special characters can be obtained. The skeletons of all special characters are first trained on the OCR text recognition model. The trained OCR text recognition model can be more adaptable to the drawing recognition in this field and improve the recognition efficiency and accuracy.

[0051] (5) Finally, each cell has corresponding text content and row and column numbers. The present invention combines the table area detection model with the OCR model to identify the positions of the cell and the text box respectively. By calculating the position of the center point of the table and the position of the center point of the cell, the matching relationship between the text box and the cell can be accurately restored. Therefore, not only can the table be identified separately, but also the text box can be identified, and the two can be matched through the position relationship. By naming the row and column numbers of the cells according to certain rules, the row and column numbers of each cell are finally obtained, which facilitates the reading of the table at work.

[0052] Example 2

[0053] A drawing table recognition method, comprising:

[0054] Input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell;

[0055] Input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box;

[0056] Calculate the distance between the center point coordinates of each text box and the center point coordinates of each cell in turn, match the text box and cell with the smallest distance between the center point coordinates of the text box and the cell, and finally fill the text content in the text box into the cell that matches the text box.

[0057] Sort the cells vertically and horizontally according to the vertex coordinates of each cell to get the row number of each cell, and calculate the number of columns of all rows. The maximum column number of the row containing the maximum number of columns is used as the maximum column number for column naming, and the column number of the corresponding cell is determined according to the vertex coordinates of each cell. If the maximum column number of the row containing the maximum number of columns is not used as the maximum column number for column naming, then if a table is edited and only 8 columns are edited, but the second row must have 10 columns to fit, then the second row is put into the table, and the 9th and 10th columns will be squeezed into the 8th column of the table.

[0058] (1) The specific method for calculating the number of columns of all rows is as follows:

[0059] Calculate the number of cells in each row, which is equal to the number of columns in each row. By arranging the cells vertically according to the coordinates, the vertical and horizontal positional relationship of the cells in the drawing is restored, so the number of rows and columns in the table can be obtained.

[0060] (2) The table area detection model is the Cycle-CenterNet model. This model solves the problems of inaccurate geometric prediction of drawings with extreme physical distortion and the defects in extracting the logical structure of misaligned tables. The Cycle-CenterNet method has significant advantages over other methods in table structure parsing. Using the TEDS metric evaluation, the accuracy of Cycle-CenterNet has been improved by 24.6%.

[0061] The OCR model is a CRNN model. The CRNN model supports the recognition of indefinite length text, has the ability to recognize words and scene text, has strong generalization ability, can be well generalized to new and unseen images, can learn end-to-end without manual feature extraction or other preprocessing steps, is robust to image distortion, and improves the practicality of the model in actual work applications.

[0062] The above Cycle-CenterNet model and CRNN model can still accurately recognize a large number of distorted drawings and tables in actual work, and have excellent robustness, which is very important in actual work.

[0063] (3) The specific method for calculating the center point coordinates of each cell according to the vertex coordinates of each cell is:

[0064] After inputting the drawing into the table area detection model, the coordinates of the four vertices of the table area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x1, y1) (x2, y2) (x3, y3) (x4, y4). The coordinates of the center point of the table area are obtained based on the coordinates of the four vertices of the table area.

[0065] In the text area detection module, the specific method for calculating the center point coordinates of each text box according to the vertex coordinates of each text box is:

[0066] After the drawing is input into the OCR model, the coordinates of the four vertices of the text box area are obtained. The upper left corner is the first point, and the coordinates of each point are output in clockwise order, which are (x ′ 1,y1 ′ )(x ′ 2,y2 ′ )(x ′ 3,y3 ′ )(x ′ 4,y4 ′ ), and get the center point coordinates based on the coordinates of the four vertices of the text box area.

[0067] In actual work, tables and text boxes in drawings are generally rectangular, so you only need to get the four vertices of the rectangle to get the coordinates of the center point by calculation.

[0068] (4) Before the drawing is input into the OCR model, the OCR model is trained using special characters. The specific method is as follows:

[0069] Special symbols in various drawings are collected, and ttf font files are drawn based on the special symbols. The ttf font files are font files of the drawing software used to make the drawings to which the special symbols belong. Binary data is generated according to the ttf font files, and drawings are generated based on the binary data. The drawings are expanded and eroded using the OpenCV tool library, and all the expanded and eroded drawings are combined to obtain a training data set. The OCR text recognition model is trained based on the training data set.

[0070] The software used in the design of each drawing is different, and the font files corresponding to the steel bar characters and other special characters used by each software are also different. The same character may be fat, thin, tall, or short, so corresponding ttf font files will be produced based on different drawing software to improve versatility.

[0071] In the construction industry, there are a large number of special characters. By using the OpenCV tool library to expand and erode the drawings, the skeletons of various special characters can be obtained. The skeletons of all special characters are first trained on the OCR text recognition model. The trained OCR text recognition model can be more adaptable to the drawing recognition in this field and improve the recognition efficiency and accuracy.

[0072] (5) Finally, each cell has corresponding text content and row and column numbers. The present invention combines the table area detection model with the OCR model to identify the positions of the cell and the text box respectively. By calculating the position of the center point of the table and the position of the center point of the cell, the matching relationship between the text box and the cell can be accurately restored. Therefore, not only can the table be identified separately, but also the text box can be identified, and the two can be matched through the position relationship. By naming the row and column numbers of the cells according to certain rules, the row and column numbers of each cell are finally obtained, which facilitates the reading of the table at work.

[0073] Example 3

[0074] A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the drawing table recognition method in Example 2 is implemented.

[0075] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0076] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. A drawing table recognition system, characterized in that: include: The drawing coordinate acquisition module is used to input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell; The text area detection module is used to input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box; The text box and cell matching module is used to calculate the distance between the center point coordinates of each text box in the drawing and the center point coordinates of each cell; match the text box with the cell whose center point coordinates are closest to the center point coordinates of the cell, and fill the text content in the text box into the cell that matches the text box.

2. The drawing table recognition system according to claim 1, characterized in that: Also includes: The table structure analysis module is used to sort each cell vertically and horizontally according to the vertex coordinates of each cell, obtain the row number of each cell, and calculate the number of columns of all rows. The maximum column number of the row containing the maximum number of columns is used as the maximum column number for column number naming, and the column number of the corresponding cell is determined according to the vertex coordinates of each cell.

3. The drawing table recognition system according to claim 2, characterized in that: The specific method for calculating the number of columns of all rows is as follows: Count the number of cells in each row, where the number of cells in each row is equal to the number of columns in each row.

4. The drawing table recognition system according to claim 1, characterized in that: The table area detection model is a Cycle-CenterNet model.

5. The drawing table recognition system according to claim 1, characterized in that: The OCR model is a CRNN model.

6. The drawing table recognition system according to claim 1, characterized in that: In the drawing coordinate acquisition module, the specific method for calculating the center point coordinates of each cell according to the vertex coordinates of each cell is: After inputting the drawing into the table area detection model, the coordinates of the four vertices of the table area are obtained. The upper left corner is the first point, and the coordinates of each point are output in a clockwise order, which are (x1, y1) (x2, y2) (x3, y3) (x4, y4). The coordinates of the center point of the table area are obtained based on the coordinates of the four vertices of the table area. In the text area detection module, the specific method for calculating the center point coordinates of each text box according to the vertex coordinates of each text box is: After inputting the drawing into the OCR model, the coordinates of the four vertices of the text box area are obtained. The upper left corner is the first point, and the coordinates of each point are output in a clockwise order, which are (x′1, y′1)(x′2, y′2)(x′3, y′3)(x′4, y′4). The coordinates of the center point are obtained based on the coordinates of the four vertices of the text box area.

7. The drawing table recognition system according to claim 1, characterized in that: The text region detection module is also used to train the OCR model with special characters before the drawing is input into the OCR model. The specific method is as follows: Special symbols in various drawings are collected, and ttf font files are drawn based on the special symbols. The ttf font files are font files of the drawing software used to make the drawings to which the special symbols belong. Binary data is generated according to the ttf font files, and drawings are generated based on the binary data. The drawings are expanded and eroded using the OpenCV tool library, and all the expanded and eroded drawings are combined to obtain a training data set. The OCR text recognition model is trained based on the training data set.

8. A drawing table recognition method, characterized in that: include: Input the drawing into the table area detection model, obtain the vertex coordinates of each cell in the drawing, and calculate the center point coordinates of each cell according to the vertex coordinates of each cell; Input the drawing into the OCR model, obtain the vertex coordinates and text content of each text box in the drawing, and calculate the center point coordinates of each text box according to the vertex coordinates of each text box; Calculate the distance between the center point coordinates of each text box and the center point coordinates of each cell in turn, match the text box and cell with the smallest distance between the center point coordinates of the text box and the cell, and finally fill the text content in the text box into the cell that matches the text box.

9. The drawing table recognition method according to claim 8, characterized in that: Also includes: The cells are sorted vertically and horizontally according to the vertex coordinates of each cell to obtain the row number of each cell, and the number of columns of all rows is calculated. The row containing the largest number of columns is used as the basis for naming the column numbers to obtain the column number of each cell.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the drawing table recognition method described in claim 8 or 9 is implemented.

Images