Method and system for extracting and displaying laboratory PDF (Portable Document Format) file table

Abstract

The invention provides a laboratory PDF file table extraction and display method and system, and the method comprises the steps: carrying out the OCR recognition and extraction of a PDF table interface in a selected region, and forming a standard data format; calculating the accuracy of each standard data item under each field based on the editing distance, the LCS length and the text length, and counting the average accuracy of all the standard data items in each row according to the unique attribute field; the standard data format is automatically presented in a new form in the LIMS after being converted and transmitted, and the average accuracy is correspondingly displayed in the last column of each row in a percentage form. According to the technical scheme, data standardization and process automation are achieved through seamless connection with an LIMS system, manual intervention is reduced, and working efficiency is improved; and real-time monitoring and evaluation functions can be realized by establishing accuracy calculation of the extracted data, so that a user can know a data processing state in time, and controllable quality is ensured.

Description

Technical Field

[0001] The present application belongs to the technical field of PDF table extraction, and specifically relates to a method and system for extracting and displaying laboratory PDF file tables. Technical Background

[0002] Traditional laboratory data management mainly relies on manual data entry and maintenance, which has many disadvantages. First, manual data entry is not only inefficient but also prone to errors, making it difficult to ensure data quality. Secondly, faced with a large number of data updates, the cost of manual maintenance is high, and it is difficult to achieve real-time data updates and traceability. In addition, data exchange between different laboratories and between laboratories and information systems also has problems such as inconsistent formats and poor compatibility, which further increases the difficulty of data management.

[0003] With the continuous in-depth application of OCR (optical character recognition) technology, the efficiency of extracting table data from PDF documents has been significantly improved. These solutions usually use OCR technology to identify text information in PDF documents, and use image processing and pattern recognition algorithms to locate and extract table data. Then, the extracted table data is converted into a standardized format, such as CSV or Excel files, through a data conversion algorithm for subsequent data processing and analysis. However, the data extracted by existing OCR technology is often not processed in any way. On the one hand, it cannot meet the standard laboratory management system data structure, and on the other hand, it cannot indicate whether the data is accurate. Therefore, it is even more impossible to display the accuracy of the extracted data in real time or realize secondary confirmation of the visual interface.

[0004] Technical content

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method and system for extracting and displaying laboratory PDF file tables, which realizes the possibility of data standardization, real-time display and secondary revision by calculating the accuracy of the extracted data and seamlessly connecting with the LIMS system.

[0006] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0007] In a first aspect, an embodiment of the present invention provides a method for extracting and displaying a table in a laboratory PDF file, comprising:

[0008] Recognize and extract the PDF table interface in the selected area through OCR to form a standard data format;

[0009] The accuracy of each standard data item under each field is calculated based on the edit distance, LCS length and text length, and the average accuracy of all standard data items in each row is calculated based on the unique attribute field;

[0010] After conversion and transmission, the standard data format is automatically presented in a new table in the LIMS system and the average accuracy is correspondingly displayed in the last column of each row in percentage form.

[0011] Optionally, the calculation of the accuracy of each standard data item under each field based on the edit distance, LCS length and text length further includes: adjusting the accuracy evaluation according to the resolution or pixel density of the PDF; the calculation formula of the accuracy percentage A is:

[0012]

[0013] Among them, |S 1 |First identify the text length for a standard data item, |S 2 | is the length of the secondary identification text of the standard data item, d(S 1 , S 2 ) is S 1 and S 2 The edit distance between 1 , S 2 ) is S 1 and S 2 The length of the longest common subsequence, α is the weight factor.

[0014] Optionally, the average accuracy is presented by different colors from 0% to 100%.

[0015] Optionally, the standard data format is automatically presented in a new form in the LIMS system after conversion and transmission, and also includes: automatically filling in non-critical system fields related to the unique attribute field, including version number, unit and value type.

[0016] Optionally, each cell of the new table supports manual modification, and after each modification, the average accuracy corresponding to the cell is automatically recalculated and updated for display, and the LIMS system saves the entry.

[0017] In a second aspect, an embodiment of the present invention further provides a system for extracting and displaying a table of a laboratory PDF file, comprising:

[0018] Extraction module: used to recognize and extract the PDF table interface in the selected area through OCR to form a standard data format;

[0019] Calculation module: used to calculate the accuracy of each standard data item under each field based on the edit distance, LCS length and text length, and to count the average accuracy of all standard data items in each row based on the unique attribute field;

[0020] Transmission and display module: used to automatically present the standard data format in a new table in the LIMS system after conversion and transmission, and the average accuracy is correspondingly displayed in the last column of each row in the form of percentage.

[0021] Optionally, the system further comprises: an adjustment module, configured to support manual modification of each cell of the new table, and after each modification, the average accuracy corresponding to the cell is recalculated by the calculation module and updated for display.

[0022] Optionally, the calculation module is further configured to adjust the accuracy assessment according to a resolution or a pixel density of the PDF.

[0023] This technical solution simplifies the directional extraction process of the content in the PDF file through a visual interface, reduces the difficulty of operation, and improves the convenience of operation for non-technical personnel. The directional extraction through manually selected areas ensures the accuracy of the data, and the built-in intelligent processing function automatically corrects common errors, further improving the data quality. Seamless connection with the LIMS system realizes data standardization and process automation, reduces manual intervention, and improves work efficiency. By establishing the accuracy calculation of the extracted data, real-time monitoring and evaluation functions can be realized, allowing users to understand the data processing status in a timely manner to ensure quality control. In addition, the function of supporting manual secondary correction enhances the flexibility of the system and data accuracy, and provides additional protection for processing complex or special format documents. In summary, the present invention has achieved significant improvements in data processing efficiency and accuracy.

[0024] Instruction Manual

[0025] Figure 1 A schematic flow chart of a method for extracting and displaying a table from a laboratory PDF file provided by an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of an interface for selecting a table of a PDF file to be extracted in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the table interface presented by the LIMS system after conversion in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0029] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0030] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] Reference Figure 1 The embodiment of the present invention provides a method for extracting and displaying a table in a laboratory PDF file, which specifically includes:

[0033] Step S1: Recognize and extract the PDF table interface in the selected area through OCR to form a standard data format.

[0034] Specifically, refer to the attached Figure 2 After the operator selects the content in the PDF file through the canvas mode, he / she can use the "Extract Quality Standard Button" provided by the interface to parse the data in the area through OCR in the back-end to form a standard data format. The standard data format in this embodiment refers to supporting one-dimensional arrays and tables to JSON format. Figure 2 Take the selected table content as an example to explain:

[0035] The parsed one-dimensional array is:

[0036] ["project", "excellent product", "qualified"]

[0037] ["Propylene oxide, w / %", ">=99.95", ">=99.80"]

[0038] ["color", "<=5", "<=10"]

[0039] The converted JSON format is:

[0040]

[0041] Step S2: Calculate the accuracy of each standard data item under each field based on the edit distance, LCS length and text length, and calculate the average accuracy of all standard data items in each row according to the unique attribute field.

[0042] In particular, this embodiment also adjusts the accuracy evaluation according to the resolution or pixel density of the PDF, and the accuracy is displayed as a percentage A, and the calculation formula is:

[0043]

[0044] Among them, |S 1 |First identify the text length for a standard data item, |S 2 | is the length of the secondary identification text of the standard data item, d(S 1 , S 2 ) is S 1 and S 2 The edit distance between 1 , S 2 ) is S 1 and S 2 The length of the longest common subsequence of the LCS, ResolutionFactor (R) is designed to return a value close to 1 or slightly greater than 1 when the resolution is high (to enhance accuracy), and a value slightly less than 1 when the resolution is low (to reduce accuracy). α is a weight factor used to balance the impact of LCS length and edit distance on accuracy. The value of α can be adjusted according to the specific application scenario. When α is large, the edit distance has a greater impact on accuracy; when α is small, the LCS length has a greater impact on accuracy.

[0045] The specific application of this formula is as follows:

[0046] For example, in the data extracted from a table, OCR first recognizes text S 1 :Water content 0.71; OCR secondary recognition text S 2 : Water content 0.11.

[0047] Light condition L: good, such as L = 0.9; Resolution R: good, such as R = 0.95; α is set to 0.5, that is, the relative importance of LCS length and edit distance in accuracy evaluation is equal.

[0048] Setting function LightBoost(L): When the light condition is good, the system obtains the corresponding value 1.1 according to the light reference table;

[0049] Set the function LightPenalty(L): When the light conditions are good, this function returns a value close to 1, such as 1.01;

[0050] Set the function ResolutionBoost(R): for good resolution, this function should return a value greater than 1, such as 1.2;

[0051] Set the function ResolutionPenalty(R): Due to the high resolution, this function should return a value close to 1, such as 1.02

[0052] LCS(S 1 , S 2 ): length is 5. The length of the longest common subsequence is "water content 0." Here, because LCS generally considers continuous matching of characters, this example ignores the difference after the decimal point and considers that the LCS is "water content 0." in "water content 0.7" and "water content 0.1", with a length of 5.

[0053] d(S 1 , S 2 ): The edit distance is 1 because there is only one character different, namely "7" and "1".

[0054] Assuming that the string length includes spaces and percent signs, min(|s1|,|s2|)=min(11,11)=11.

[0055] Substituting into the calculation A=((11*5*1.1*1.2 / 110)+0.5*(1-11*1*1.01*1.02 / (11*1)))*100=77.69%, so the accuracy of the standard data item identified this time is 77.69%.

[0056] Similarly, according to this calculation method, the accuracy of each standard data item corresponding to each field is calculated one by one, and then the average accuracy is calculated. Figure 2 Take the table selected in as an example, calculate the accuracy of all standard data items such as "propylene oxide" and "color" under the "project" field, calculate the accuracy of standard data items such as ">=99.95" and "<=5" under the "superior product" field, and calculate the accuracy of standard data items such as ">=99.80" and "<=10" under the "qualified product" field; then, since the only attribute field here is "project", for example, the average accuracy of the row where propylene oxide is located is: the sum of the accuracy of "propylene oxide" + the accuracy of ">=99.95" + the accuracy of ">=99.80" divided by the number of items 3. Calculate the others in the same way.

[0057] Step S3: After the standard data format is converted and transmitted, it is automatically presented in a new table in the LIMS system and the average accuracy is correspondingly displayed in the last column of each row in percentage form.

[0058] For details, please refer to the attached Figure 3 ,for Figure 2 Select the table style to be displayed in the LIMS system after the table conversion. The last column of the new table shows the average accuracy of each row of data, and is presented in different colors from 0% to 100%. For table rows with low accuracy, the erroneous data can be adjusted by manually revising the corresponding cells twice, and then saved and entered into the LIMS system. After each modification, the average accuracy corresponding to the cell is automatically recalculated and updated. In particular, some non-critical system fields related to the unique attribute field can also be automatically filled in the generated new table, such as the version number, unit and value type in this embodiment.

[0059] In addition, an embodiment of the present invention also provides a system for extracting and displaying a table of a laboratory PDF file, including:

[0060] Extraction module: used to recognize and extract the PDF table interface in the area selected in canvas mode through OCR to form a standard data format.

[0061] Calculation module: used to calculate the accuracy of each standard data item under each field based on the edit distance, LCS length and text length, and to count the average accuracy of all standard data items in each row according to the unique attribute field; specifically, the calculation module is also used to adjust the accuracy evaluation according to the resolution or pixel density of the PDF.

[0062] Transmission and display module: used to automatically present the standard data format in a new table in the LIMS system after conversion and transmission, and the average accuracy is displayed in the last column of each row in percentage.

[0063] The adjustment module is used to support manual modification of each cell of the new table, and after each modification, the average accuracy corresponding to the cell is recalculated by the calculation module and updated for display.

[0064] The technical effect that can be achieved by this embodiment is basically the same as the technical effect that can be achieved by the aforementioned method for extracting and displaying a table from a laboratory PDF file, and will not be described in detail here.

[0065] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0066] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0068] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for extracting and displaying a table from a laboratory PDF file, characterized in that include: Recognize and extract the PDF table interface in the selected area through OCR to form a standard data format; The accuracy of each standard data item under each field is calculated based on the edit distance, LCS length and text length, and the average accuracy of all standard data items in each row is calculated based on the unique attribute field; After conversion and transmission, the standard data format is automatically presented in a new table in the LIMS system and the average accuracy is correspondingly displayed in the last column of each row in percentage form.

2. A method for extracting and displaying a laboratory PDF file table as claimed in claim 1, characterized in that The calculation of the accuracy of each standard data item under each field based on the edit distance, LCS length and text length also includes: adjusting the accuracy evaluation according to the resolution or pixel density of the PDF; the calculation formula of the accuracy percentage A is: Among them, |S1| is the length of the first recognized text of a certain standard data item, |S2| is the length of the second recognized text of the standard data item, d(S1, S2) is the edit distance between S1 and S2, LCS(S1, S2) is the length of the longest common subsequence of S1 and S2; α is the weight factor.

3. A method for extracting and displaying a laboratory PDF file table as claimed in claim 1, characterized in that The average accuracy is represented by different colors from 0% to 100%.

4. A method for extracting and displaying a table from a laboratory PDF file as claimed in claim 1, characterized in that The standard data format is automatically presented in a new form in the LIMS system after conversion and transmission, and also includes: Automatically populate non-critical system fields related to the unique attribute field, including version number, unit, and value type.

5. A method for extracting and displaying a laboratory PDF file table as described in any one of claims 1 to 4, characterized in that Each cell of the new table supports manual modification, and after each modification, the average accuracy corresponding to the cell is automatically recalculated and updated, and the LIMS system saves the entry.

6. A system for extracting and displaying tables from laboratory PDF files, characterized in that include: Extraction module: used to recognize and extract the PDF table interface in the selected area through OCR to form a standard data format; Calculation module: used to calculate the accuracy of each standard data item under each field based on the edit distance, LCS length and text length, and to count the average accuracy of all standard data items in each row based on the unique attribute field; Transmission and display module: used to automatically present the standard data format in a new table in the LIMS system after conversion and transmission, and the average accuracy is correspondingly displayed in the last column of each row in the form of percentage.

7. A system for extracting and displaying a table from a laboratory PDF file as described in claim 6, characterized in that The system also includes: an adjustment module, which is used to support manual modification of each cell of the new table, and after each modification, the average accuracy corresponding to the cell is recalculated by the calculation module and updated for display.

8. A system for extracting and displaying a table from a laboratory PDF file as described in claim 6 or 7, characterized in that The calculation module is further configured to adjust the accuracy evaluation according to the resolution or pixel density of the PDF.

Images