Data extraction system

Abstract

The invention relates to data management. To efficiently and reliably extract data from one or more tabular data files, a data extraction system is provided for extracting data from one or more tabular data files. The system comprises a user interface adapted to facilitate one or more users to submit one or more tabular data files, each tabular data file comprising at least one table. According to a first alternative, the user interface is configured to receive a user-defined template comprising at least one target table. The data extraction system further comprises a data extraction module or a validator. The data extraction module comprises a schema-level matcher and an instance-level matcher. The schema-level matcher is configured to identify target schema-level information of the at least one target table and to select at least one semantically matching candidate table from the submitted one or more tabular data files based on the target schema-level information. The instance-level matcher is configured to identify target instance-level information of the at least one target table and to extract data from the at least one semantically matching candidate table based on the target instance-level information. According to a second alternative, the user interface is configured to receive a validator template. The data extraction system further comprises a validator configured to apply the validator template to at least one table of the one or more tabular data files to validate a table format of the at least one table based on rules for labels, metadata and / or data defined by the validator template.

Description

Technical Field

[0001] This invention relates to data management. In particular, it relates to a data extraction system for extracting data from one or more tabular data files, a distributed database system, a computer-implemented method for extracting data from one or more tabular data files, a computer program element, and a computer-readable medium. Background Technology

[0002] In many environments, such as laboratory, production, or business-related settings, large amounts of data may be collected from a variety of sources. For example, in a laboratory setting, documents may originate from numerous experimental tests conducted in the past using various laboratory equipment, with much of the data captured, recorded, and exchanged using tabular data files (such as Excel files). This could generate hundreds to thousands of such documents for each project.

[0003] Since many of these files already contain tabular data structures, automatic extraction is usually straightforward, as the structure remains consistent across all files in a given project. In this case, the data can be easily migrated to any other system for data analysis. However, once this structure deviates even slightly within a file, data extraction can become a more difficult and often time-consuming process, potentially requiring rigorous manual data structuring and cleaning. Summary of the Invention

[0004] It may be necessary to extract data efficiently and reliably from one or more tabular data files.

[0005] The object of the invention is solved by the subject matter of the independent claims, wherein further embodiments are included in the dependent claims. It should be noted that the aspects described below also apply to data extraction systems for extracting data from one or more tabular data files, distributed database systems, computer-implemented methods for extracting data from one or more tabular data files, computer program elements, and computer-readable media.

[0006] According to a first aspect of the invention, a data extraction system is provided for extracting data from one or more tabular data files. The system includes a user interface adapted to facilitate submission of one or more tabular data files by one or more users, each tabular data file including at least one table.

[0007] According to the first alternative, the user interface is configured to receive a user-defined template including at least one target table. The data extraction system also includes a data extraction module. The data extraction module includes a schema-level matcher and an instance-level matcher. The schema-level matcher is configured to identify target schema-level information of the at least one target table and select at least one semantically matching candidate table from one or more submitted tabular data files based on the target schema-level information. The instance-level matcher is configured to identify target instance-level information of the at least one target table and extract data from the at least one semantically matching candidate table based on the target instance-level information.

[0008] According to the second alternative, the user interface is configured to receive a validator template. The data extraction system also includes a validator configured to apply the validator template to at least one table in the one or more tabular data files to validate the table format of the at least one table based on rules defined by the validator template for tags, metadata, and / or data.

[0009] In other words, a hybrid matcher can be provided to directly combine pattern matching and instance matching to extract tables from a series of tabular files (e.g., Excel files) based on a user-defined template. One or more tabular data files may include data obtained from at least one of the following: laboratory environment, production environment, and business-related environment. For example, one or more submitted tabular data files may include experimental data obtained from a laboratory environment. Experimental data may include data generated through measurement, test methods, experimental designs, or quasi-experimental designs. In another example, one or more tabular data files may include business-related data (e.g., financial or control). The user-defined template defines at least one target table. To extract the table, attributes of at least one target table are determined, which serve as an information source to provide two types of information, including schema-level information and instance-level information. Schema-level information relates to the inherent attributes of the table itself, which each candidate must satisfy, such as name, description, data type, relationship type (part-of, is-a, etc.), constraints, and schema structure. Schema elements of one or more submitted tabular data files can be mapped to the target table of the user-defined template. For example, simple scalar relations (e.g., =, <), functions (e.g., addition or concatenation), entity relations (e.g., is-a, part-of) or set-oriented relations (e.g., overlap, containment) can be used. Pattern-level information is used to exclude semantically mismatched candidate tables and their respective transformations. Instance-level information relates to attributes belonging to the data, which are likely to be different for each input table, such as data values. Instance-level information can be used to assign scores to semantically matching candidate tables based on similarity measures and to extract data with the desired scores from semantically matching candidate tables. Entity matching or record linking can be implemented as part of the instance-level matcher, which serves as a measure for comparing tables. Entity matching aims to identify records that reference the same entity. Entity matching can also be used to identify tables similar to reference tables. For example, data extraction can work as follows: i) a sample file is loaded, and the user (e.g., the person who originally created the data) selects a sample for each type of table he / she wants to extract, for example, through a graphical user interface. This collection of sample tables can also be called a user-defined template. This will be discussed in particular below. Figure 2 The exemplary embodiments described herein will be explained. ii) The data extraction module scans all other files belonging to the project and extracts the most promising candidate tables. This will be explained below, and particularly regarding... Figure 1A The exemplary embodiments described herein will be explained. iii) Following the verification step performed by the user, for example by extracting a graphical report of the results, all found and accepted tables are loaded into the database. This will be explained below and in particular regarding Figure 4The exemplary embodiments described herein will be explained. iv) The user can then select the primary key for each table type in the database, based on which a model or schema is automatically generated. These selections can be updated at any time to allow for flexible schemas and refactoring as needed. The schema, along with its primary key, foreign key, and unique key, is displayed graphically in the user interface. This can also be viewed as a virtual whiteboard, as it not only provides a good overview of the database structure but also serves as an effective communication tool between different teams by highlighting key data join points.

[0010] Alternatively, validators can be provided to validate table formats based on validator template formats, thereby achieving a degree of consistency in early data collection. Validator templates define rules for tags, metadata, and / or data. For example, each individual Excel sheet can be considered a separate and irreducible data source, i.e., a minimal structure. Within each Excel sheet, tags define the data and metadata information within the sheet. Examples of tags include, but are not limited to, the HEADER tag, KEY_TYPE tag, KEY_SUFFIX tag, UNITS tag, QUALIFIER tag, NOTES tag, FORMAT tag, TYPE tag, and DATA tag. For example, the HEADER tag can include, but is not limited to, mandatory tags, identifier names / labels such as primary and foreign keys and attributes, label names and naming conventions, and blank headers. The KEY_TYPE tag is used to create logical connections between different sheets, including one-to-one or one-to-many relationships. The DATA tag is used to indicate that a row or column contains data points. The FORMAT tag defines the basic data type of the data values ​​in the DATA section. This can include quantities (e.g., integers, real numbers, strings in scientific notation), strings, and dates (Excel time (5 digits) or dd / mm / yyyy). The UNITS tag contains the units of the data, preferably SI units. The NOTES tag is an additional, human-readable extension to the header. Different rules can be applied to each tag. For example, rules for the HEADER tag may include, but are not limited to: i) the header label must not contain any special characters; ii) the header label must begin with a character; iii) the header label must not contain any spaces; iv) the header label must not contain any hyphens and should be in lowercase; v) the header name of the primary key must be unique within the data form; and vi) there is no primary key that matches a foreign key. Rules for the DATA tag may include, but are not limited to: i) data field values ​​(QUANTITY) must contain only significant numbers; ii) missing values; iii) invalid dates in dd / mm / yyyy format; and iv) the identifier of the primary key DATA is not globally unique. An example rule for the KEY_TYPE tag is that the primary key type is allowed only once per form. For example, a validator might work as follows: i) Analyze a collection of files containing tabular data structures and, based on the analysis of the collection, define rules for tags, metadata, and / or data, for example, by a designated data scientist. ii) Derive a validator template from the rules to validate a given table for, for example, the format of tags, numbers, data, and time; the uniqueness of identifiers and attributes; the completeness and uniqueness of metadata; and the logical structure based on primary and foreign keys. iii) Then apply the validator template to the original files and reject any files that violate the criteria described in ii).

[0011] Both the data extraction module and the validator provide a simple and fast method for data aggregation, thereby enabling more robust, broader, and more reliable data analysis, such as in distributed laboratory systems.

[0012] In particular, efficiency can be improved by using a data extraction module, as semantically mismatched candidates can be filtered out early on, and candidates that only semantically match can be further evaluated and assigned a score indicating their similarity to tables in the template. Furthermore, unlike purely index-based methods, the use of schema-level and instance-level matchers can be applied to another file, even if the tables of interest are not in exactly the same location, such as table relocation, transposed tables, or additional table columns. Therefore, a smaller number of templates may be needed to successfully parse an entire project and integrate it into a well-defined database structure. This can be beneficial for tabular files collected from different sources (e.g., from different labs) using different Excel sheets in a distributed lab system. Since tabular files from different sources may have different formats, proper data analysis may not be possible. Hybrid matchers can allow the collection and cleaning of such uneven data, thus allowing for more robust, extensive, and reliable data analysis. Moreover, as will be explained below, hybrid matchers can also use machine learning to provide access to large amounts of historical data. This can provide automated support for data analysis, making it easier to build application-specific tools that include automated data extraction.

[0013] By using a validator, the table format of submitted tabular data files is validated based on a template format to achieve a degree of consistency during early data collection. In some examples, a validator template might be defined for, for example, an Excel spreadsheet from a single laboratory. In other examples, the validator template might vary between different laboratories. Inconsistencies can be reported to the user graphically, for example through a user interface, or automatically corrected. Thanks to this early consistency step, submitted tabular data files can be automatically constructed, reducing any data searching, cleaning, and validation work when they need to be accessed, for example, for machine learning or other data-driven modeling. This also allows for a higher success rate in finding related data files across, for example, different laboratories or different production units.

[0014] According to an embodiment of the present invention, an instance-level matcher is configured to determine a similarity metric between at least one semantic matching candidate table and at least one target table, and to extract data from at least one semantic matching candidate table whose similarity metric exceeds a given threshold.

[0015] For example, an instance-level matcher can be a constraint-based matcher that determines corresponding matching candidates based on constraint-based features such as numerical ranges and averages or character patterns. In another example, instance-level matching can be performed using per-instance matching. For instance, a record linking process can be used to identify records in a dataset that are linked to the same entity. This can be done by defining a similarity metric between all records and then assigning any record whose similarity exceeds a certain threshold to the same entity.

[0016] According to an embodiment of the present invention, a similarity measure is assigned to each column of the target table in a cascading manner: i) a similarity measure is specified for each column of the target table; ii) a similarity measure is specified for all columns of a specific type; and iii) all remaining columns are set to the default similarity, which is optionally user-defined.

[0017] For example, if the user has not covered any content, the default similarity metric is the Jaro-Winkler distance, which is a string metric that measures the edit distance between two sequences.

[0018] According to an embodiment of the present invention, a pattern-level matcher is configured to transform at least one table of one or more submitted tabular data files into at least one transformed table to match the format defined by at least one target table and select at least one semantically matching candidate table from the at least one transformed table.

[0019] Because users are free to choose the layout that best suits their needs, there may be a list of inconsistencies in the submitted tabular data files, such as changed table positions, transposed tables, additional columns added in between, and so on. Transformations can be used to transpose tables, extract larger sub-tables, remove redundant columns, and / or replace remaining columns. This transformation can resolve the inconsistency issues between submitted tabular data files, thereby reducing any data search, cleaning, and validation work when the submitted tabular data files need to be accessed, such as for machine learning or other data-driven modeling.

[0020] According to an embodiment of the present invention, the transformation includes at least one of the following: extracting a sub-table from at least one table, transposing at least one table, replacing columns of at least one table, and removing at least one column or row from at least one table that contains mismatched schema-level information.

[0021] According to an embodiment of the invention, a pattern-level matcher is configured to perform a sequence of operations on at least one table of one or more submitted tabular data files. This sequence of operations defines a tree structure with multiple edges and multiple nodes, where each edge corresponds to a basic transformation of a parent node having one or more free parameters, and each node corresponds to a transformation table of the corresponding basic transformation. The pattern-level matcher is configured to select at least one transformation table at multiple leaf nodes of the tree structure, where the similarity between this at least one transformation table and at least one target table is higher than the similarity between the remaining transformation tables and at least one target table. The pattern-level matcher is configured to select at least one semantically matching candidate table from the at least one selected transformation table.

[0022] For transformations, there may be many free parameters, such as the indexes of columns to be deleted / transposed, and thus the number of candidate tables generated can be large. Finding a data-manipulating transformation that satisfies a given set of user requirements, i.e., a user-defined template, can be viewed as a search in a tree. Therefore, a series of operations may be needed to determine the free parameters, such as column indices. This defines a tree structure where each edge corresponds to a basic transformation with certain free parameters, and each node corresponds to the result of the corresponding transformation. Each leaf node created, i.e., the outermost or terminal node of the tree structure, adheres to at least one schema attribute of the target table (e.g., the data type of the column). Otherwise, the edges leading to it would be pruned earlier. Similarity can only be used to compare leaf nodes with at least one target table; that is, only the data attributes of the leaf nodes, i.e., instance-level information, can be compared with at least one target table. This can be done by applying the actual values ​​of each in the cells of interest to establish a concept of similarity between the transformed candidate tables and at least one target table. Transformation tables can be ordered by similarity. The top-ranked transformation tables can be used for instance-level matching. This can be beneficial in reducing the number of semantic matching tables for subsequent instance-level matchers, as schema-level information can be used to prune many branches early.

[0023] According to embodiments of the present invention, a pre-trained machine learning classifier is provided, which is suitable for performing pattern matching and / or instance matching. The pre-trained machine learning classifier is configured to be trained using a sample set of tabular data files, the sample set of tabular data files having representative data files for parsing tabular data files, a set of successfully parsed tabular data files, and a set of failed tabular data files.

[0024] In other words, pre-trained machine learning classifiers synthesize transformations based solely on output samples (i.e., not input-output sample pairs) so that the user's workload remains constant regardless of the number of different layouts. Conversely, typical machine learning determines the required transformation based on input-output samples. Input-output pairs are typically located at the cell level in a form, making it usually only necessary to provide two or three samples to the frame to synthesize the correct transformation.

[0025] According to an embodiment of the present invention, the user interface is configured to provide a report including at least one of the following: an overview of extracted statistics, a response table showing the number of tables extracted from each tabular data file, form, and / or table type, and an interface allowing users to directly migrate the extracted tables to a database.

[0026] For example, the interface allows users to link and migrate extracted tables to a database. That is, information from various files can thus be collected, linked, and made searchable, making it easy for users to obtain all the data for the desired attributes. Furthermore, if needed, users can download this collection as an Excel or CSV file at any time, for example, as input to other software in the analysis pipeline. The database can then be modified or connected to other databases for analysis.

[0027] According to embodiments of the present invention, the rules for tags, metadata and / or data include at least one of the following: i) the format of tags, numbers, dates and times, ii) the uniqueness of identifiers and attributes, iii) the completeness and integrity of metadata, and iv) a logical structure based on primary keys and foreign keys.

[0028] Examples of formatting for labels, numbers, data, and time may include, but are not limited to: i) title labels must not contain any special characters, ii) title labels are not allowed (NA), iii) data field values ​​(QUANTITY) must contain only valid numbers, and iv) invalid data in the format dd / mm / yyyy. Examples of uniqueness for identifiers and attributes may include, but are not limited to: i) attribute names must be unique within the data form, and ii) primary key types are allowed only once per form.

[0029] According to an embodiment of the present invention, the validator template can be derived from a user-defined schema based on the analysis of the one or more submitted form data files.

[0030] For example, analyze a collection of files containing tabular data structures and design a schema for a database, perhaps by a designated data scientist. From this schema, derive a validator template that validates column types, key uniqueness, foreign key correctness, and header values ​​for a given table.

[0031] According to embodiments of the present invention, one or more table files have one or more formats, including at least one of the following: comma-separated value CSV format and The document is an electronic form file format. Optionally, one or more form files to be verified by the verifier have one or more formats, including Open Document Format (ODF), Lotus... Worksheet format Works electronic form format, At least one of the following: electronic form format, Abykus worksheet format, and / or Ability electronic form format.

[0032] According to embodiments of the present invention, a distributed database system is provided. The distributed database system includes one or more data repositories and a data extraction system described above and below. The one or more data repositories are configured to store tabular data files from multiple sources, each tabular data file including at least one table. The data extraction system is configured to extract data from the tabular data files based on a user-defined template including at least one target table.

[0033] Distributed database systems can be distributed laboratory systems, distributed production systems, or distributed business systems. For example, in a distributed laboratory system, one or more data repositories can be populated from different laboratories using different Excel spreadsheets. Because users are free to choose the layout to best suit their needs, multiple experiments from different sources may have different formats, which can hinder proper data analysis. The data extraction systems described above and below allow for the collection and cleaning of such data, thereby enabling more robust, extensive, and reliable data analysis. By providing access to large amounts of historical data, automated processes using, for example, machine learning and other data-driven models can be enabled.

[0034] According to a second aspect of the invention, a computer-implemented method for extracting data from one or more tabular data files is provided.

[0035] According to the first alternative, the method includes the following steps: a) receiving one or more tabular data files, each tabular data file including one or more tables; b1) receiving a user-defined template including at least one target table; b2) identifying target schema-level information of the at least one target table; b3) selecting at least one semantically matching candidate table from the submitted one or more tabular data files based on the target schema-level information; b4) identifying target instance-level information of the at least one target table; and b5) extracting data from the at least one semantically matching candidate table based on the target instance-level information.

[0036] According to a second alternative, the method includes the following steps: a) receiving one or more tabular data files, each tabular data file including one or more tables, and c) applying a validator template to at least one table in the one or more tabular data files to validate the table format of the at least one table based on the template format of the validator template.

[0037] In other words, a hybrid matcher, comprising schema-level and instance-level matchers, is used to extract data: the schema-level matcher uses schema-level information to select candidate tables (or their respective transformations) and exclude tables (their respective transformations), while the instance-level matcher uses instance-level information to assign scores to candidate tables. The instance-level matcher is built on the concept of record links, typically done by defining a similarity metric between all records and then assigning any record whose similarity exceeds a certain threshold to the same entity. This approach learns only the type of rows / columns (schema-level information) and the values ​​(instance-level information), not their exact location (not index-based), and the table's location and whether it has changed are irrelevant. Therefore, due to formatting and / or experimental modifications, this method is applicable to data files from different laboratories, production units, and business units.

[0038] Alternatively, a collection of files containing tabular data structures is analyzed, and rules for tags, metadata, and / or data are defined based on the analysis of the file collection, for example, by a designated data scientist. From the rules, a validator template is derived that validates, for example, the format of tags, numbers, data, and time; the uniqueness of identifiers and attributes; the completeness and uniqueness of metadata; and the logical structure based on primary and foreign keys. The validator template can then be applied to the original files and any files that violate the standards can be rejected. In this way, the automatic extraction of the desired data can be achieved. According to a third aspect of the invention, a computer program element is provided for the system described herein, which, when executed by a processing unit, is adapted to perform the described method steps.

[0039] According to a fourth aspect of the present invention, a computer-readable medium storing program elements is provided.

[0040] As used herein, the term "user interface" refers to an interface between a human user or operator and one or more devices that enables communication between the user and the devices. Examples of user interfaces that may be employed in various implementations of this disclosure include, but are not limited to, switches, potentiometers, buttons, dials, sliders, trackballs, displays, various types of graphical user interfaces (GUIs), touchscreens, microphones, and other types of sensors that can receive some form of human-generated stimulus and generate signals in response thereto.

[0041] As used herein, the term "module" may refer to, be a part of, or include an application-specific integrated circuit (ASIC), electronic circuit, processor (shared, dedicated, or grouped) and / or memory (shared, dedicated, or grouped), combinational logic circuit, and / or other suitable component that performs one or more software or firmware programs.

[0042] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0043] Exemplary embodiments of the present invention will now be described with reference to the following figures:

[0044] Figure 1A A block diagram of a data extraction system according to some embodiments of the present disclosure is shown.

[0045] Figure 1B A block diagram of a data extraction system according to some other embodiments of the present disclosure is shown.

[0046] Figure 2 Examples of graphical user interfaces for submitting user-defined templates according to some embodiments of this disclosure are shown.

[0047] Figure 3 Examples of transformations of the synthesis and evaluation candidate tables according to some embodiments of this disclosure are shown.

[0048] Figure 4 Examples of graphical user interfaces for viewing extraction reports according to some embodiments of this disclosure are shown.

[0049] Figure 5 Examples of distributed database systems according to some embodiments of this disclosure are shown.

[0050] Figure 6A A flowchart illustrating a computer-implemented method for extracting data from one or more tabular data files according to some embodiments of the present disclosure is shown.

[0051] Figure 6B A flowchart illustrating a computer-implemented method for extracting data from one or more tabular data files, according to another embodiment of the present disclosure, is shown. Detailed Implementation

[0052] In some cases where the structure of tabular data files is distorted, data extraction can become more difficult and time-consuming, often requiring rigorous manual data structuring and cleansing. Messy data can arise from a variety of reasons. For example, users may store one or more tables in different cell locations within a spreadsheet and have the freedom to choose their layout to best suit their work. Take laboratory experiments as another example: different labs may run similar experiments but structure their files differently. When there is no explicit template for documentation, different users may structure their files differently. Other experiments may be added to one file but not to others.

[0053] To extract data from tabular data files, especially those with messy data, the user interface is designed to directly obtain information about the file structure from the source, such as that provided by a user designing and filling out an Excel workbook. This file structure information is provided in a user-defined template. This saves time compared to additional meetings where users' data structure knowledge is transferred to data experts. The hybrid matcher eliminates semantically mismatched candidates based on the file structure information obtained from the user-defined template. Only semantically matching candidates are further evaluated to determine if they are similar to the target table in the user-defined template. This improves efficiency. Since the hybrid matcher only learns the types of rows / columns (schema-level information) and values ​​(instance-level information), not their exact location (not index-based), the table's location and whether it has changed are irrelevant. This can be advantageous for data extraction even when the structure deviates from the tabular data file. Candidates with high similarity ranking can be extracted and migrated to the database, allowing users to easily obtain all the data for the desired attributes. Alternatively, a validator can be provided to validate the table format based on a validator template format, achieving some consistency in early data collection. The validator template defines rules for tags, metadata, and / or data. By leveraging the initial consistency steps, submitted tabular data files can be automatically constructed, thereby reducing any data searching, cleaning, and validation work when they need to be accessed, for example, for machine learning or other data-driven modeling. This also allows for a higher success rate in refining associated data files across, for example, different laboratories or different production units. This will be explained in more detail in the following exemplary embodiments.

[0054] Figure 1A A block diagram illustrating an example of a data extraction system 10 for extracting data from one or more tabular data files is shown. The data extraction system 10 includes a user interface 12 and a data extraction module 14. The data extraction module 14 includes a pattern-level matcher 16 and an instance-level matcher 18.

[0055] User interface 12 is configured to facilitate the submission of one or more tabular data files 22 by one or more users, each tabular data file including at least one table 24. User interface 12 may be an interactive interface, including but not limited to a graphical user interface (GUI), a character user interface, or a touchscreen. Users may evaluate user interface 12 through electronic communication devices, including but not limited to desktop computers, laptops, mobile phones, or smartphones.

[0056] One or more tabular data files 22 may contain data of any type. Examples of data include, but are not limited to, data obtained from laboratory environments, production environments, and / or business-related environments. For example, one or more tabular data files 22 may include experimental data from one or more laboratories. For example, in a laboratory focused on the synthesis and measurement of biodegradable polymers, experimental data may include analytical data, solubility data, mechanical test data, ecotoxicity test data, and / or analytical data generated from software. Production-related data may relate to the setup in which software and other products are actually put into operation for the intended use by users. Business-related data may include data collected from, for example, financial sectors, control departments, etc. Furthermore, one or more tabular data files 22 may have various file formats, such as, but not limited to, CSV format and... The document is in electronic form format.

[0057] User interface 12 is also configured to facilitate one or more users to submit user-defined templates 26 including at least one target table 28, for example, via Figure 2 The interface shown. Figure 2 The example interface shown might resemble an Excel workbook containing a collection of labeled worksheets 32a, 32b, 32c, 32d, and 32e. Each labeled worksheet may include one or more target sheets 28. For example, Figure 2The example labeled worksheet 32a shown includes two target tables 28a and 28b. In some embodiments, the user-defined template can be a JavaScript Object Notation (JSON) file, which may contain a template name, the name of the file used to create the template, and a dictionary of endpoints for each target table within the template. The dictionary may contain not only the form name where the target tables 28a and 28b reside, but also the user-assigned table type name and its exact position in terms of indexes or endpoints, such as the start row, start column, end row, and end column. These indexes are stored separately from the headers and are used to identify the stored attributes (temperature, sample ID, etc.) and data values. Additionally, extra labels such as transpose and invert provide information about the table format. The interface shown in the example may also include options for saving their work. For example, the interface shown in the example shows a button that allows the user to add or remove tables from the user-defined template 26. Target tables 28 can be selected from an exemplary set of previously stored table data files that the user wishes to extract. In other words, the user may be allowed to upload and select tables from a single file to create target tables 28 of the user-defined template 26.

[0058] Back Figure 1A The data extraction module 14 is communicatively coupled to the user interface 12 to receive one or more tabular data files 22 and user-defined templates 26. The data extraction module 14 can be implemented in a web server. The data extraction module 14 (in the backend or server) is communicatively coupled to the user interface 12 (in the frontend or client) via a network to receive one or more tabular data files and user-defined templates.

[0059] The schema-level matcher 16 is configured to identify target schema-level information for at least one target table and select at least one semantically matching candidate table 30 from one or more submitted table data files based on the target schema-level information. The schema-level matcher 16 considers only schema information, not instance data. Schema information may include attributes of schema elements, such as name, description, data type, relation type (part-of, is-a, etc.), constraints, and schema structure. In some embodiments, schema-level information may include column types, whether columns can be nullable, and data types. Data types may include basic types, structured types, and / or custom types. Basic types may correspond to basic data types and may be totally ordered; basic types may include int, float, and str. Structured types may describe or establish a structure, including empty, unknown, and occupied types. Custom types may be subsets of defined strings, such as additional identifiers, formulas, etc. In some embodiments, the schema-level matcher 16 may utilize one or more methods to identify one or more semantically matching candidate tables. For example, pattern-level matcher 16 may include a language matcher that uses names and text including words or sentences to find semantically similar pattern elements. Name-based matching can match pattern elements with the same or similar names, such as exp. equals experiment. In some embodiments, pattern-level matcher 16 may include a description-based matcher that determines the similarity between pattern elements based on annotations accompanying the pattern elements. Typically, pattern-level matcher 16 can find multiple candidates for semantic matching. For each candidate, the degree of similarity can be estimated by normalized numerical values ​​in the range of 0 to 1 to identify suitable candidates to output to instance-level matcher 18. In terms of pattern attributes, it may only consider column type and whether a column can be null. Typically, these pattern attributes are the only information that can be reliably derived without any additional user input, apart from the user-defined template. In some embodiments, consistency of title names may not be required, as this condition may be violated to such an extent that they can be considered essentially random. In some embodiments, relatively similar or perfectly matching titles throughout the dataset may be a very strong condition, which helps to significantly simplify the transformation problem.

[0060] Instance-level matcher 18 is configured to identify target instance-level information of at least one target table 28 and extract data from at least one semantically matching candidate table 30 based on the target instance-level information. In other words, instance-level matcher 18 only considers instance data, i.e., data content. For example, the actual value of each cell in the table may be considered. The actual value of each cell may include structured data, such as numeric and string elements. For such data, constraint-based representations, such as numerical ranges and averages or character patterns, may be applied. The record linking process is used to determine records in the dataset that are linked to the same entity. For example, instance-level matcher 18 is configured to determine a similarity measure between at least one semantically matching candidate table and at least one target table, and extract data from at least one semantically matching candidate table whose similarity exceeds a reference value. In other words, this can be done by defining a similarity measure between all records and then assigning any record whose similarity exceeds a certain threshold to the same entity. Similarity measures can be assigned to each column of the target table in a cascading manner: i) specify a similarity measure for each column of the target table, ii) specify a similarity measure for all columns of a specific type, and iii) set all remaining columns to the default similarity, which is optionally user-defined. The default similarity measure can be the Jaro-Winkler distance, a string measure that measures the edit distance between two sequences. In some embodiments, the actual value of each cell of interest can be used. This information can be used to establish a concept of similarity between any tables with matching pattern attributes. For the target table, a similarity measure can be registered for each column. The default similarity is the Jaro-Winkler edit distance. Using these column similarities, the similarity between individual rows can be defined. Therefore, the similarity between two tables may simply be the average pairwise row similarity.

[0061] Optionally, the pattern-level matcher 16 can be configured to transform at least one table 24 in one or more submitted tabular data files 22 into at least one transformed table to match the format defined by at least one target table and select candidate tables from at least one semantically matching transformed table. For example, the transformation may include at least one of the following: extracting a subtable from at least one table, transposing at least one table, replacing columns of at least one table, and removing at least one column from at least one table containing mismatched pattern-level information. In other words, the transformation can be used to transpose a table, extract a larger subtable, remove redundant columns, and / or replace remaining columns. Transformation can also be referred to as data preparation. Data preparation describes the transformation of unstructured data into data with a well-structured format. Various tools have been developed to generate such transformations in a semi-automatic or fully automatic manner. All of these tools contain a pool of basic transformations that are cascaded to synthesize the desired transformation.

[0062] Figure 1B A block diagram of another example of a data extraction system 10 for extracting data from one or more tabular data files is shown. In this example, a validator 20 can be used instead of extraction module 14. Validator 20 is configured to apply a validator template to at least one table in one or more tabular data files to validate the table format of at least one table based on rules defined by the validator template for tags, metadata, and / or data. The one or more tabular files have one or more formats, including at least one of the following: comma-separated value CSV format and from Electronic form file formats, Open Document Format (ODF), Lotus Worksheet format Works electronic form format, Electronic form format, Abykus worksheet format, and Ability electronic form format. Rules for tags, metadata, and / or data include at least one of the following: i) format of tags, numbers, data, and time; ii) uniqueness of identifiers and attributes; iii) completeness and uniqueness of metadata; and iv) logical structure based on primary and foreign keys. Based on analysis of one or more submitted form data files, validator templates can be derived from a user-defined schema.

[0063] Figure 3 An example of synthesizing and evaluating transformations of candidate tables according to embodiments of the present disclosure is shown. A pattern-level matcher 16 is configured to perform a sequence of operations on at least one table 24 of one or more submitted tabular data files 22, the sequence of operations defining a tree structure 34 having multiple edges 36 and multiple nodes 38. Each edge 36 corresponds to a basic transformation of a parent node having one or more free parameters, and each node 38 corresponds to a transformation table of the corresponding basic transformation. The pattern-level matcher is configured to select at least one transformation table at multiple leaf nodes 40 of the tree structure whose similarity to (at least one) target table is higher than the similarity of the remaining transformation tables to (at least one) target table. The pattern-level matcher 16 is configured to select at least one semantically matching candidate table 30 from at least one selected transformation table. Leaf nodes 40 are the outermost nodes or terminal nodes of the tree structure 34. For example, a score representing similarity can be assigned to each leaf node 40. Figure 3In the example shown, a leaf node 40 with a score of 0.76 can be selected. The corresponding transformed candidate table is then available for data extraction. The instance-level matcher 18 is then configured to extract data from at least one selected transformed candidate table (i.e., the leaf node with a score of 0.76). In some embodiments, the score can be calculated in a manner similar to the similarity metric described above. In some embodiments, this can be accomplished by calculating at least one row in the target table that has the highest similarity relative to all other rows in that target table (also called centroid) and comparing it to those rows. Furthermore, the target table can be updated after every n successfully parsed candidate tables, where n typically increases throughout the parsing process. This process ensures that the evaluation of all transformations is valid for any use case at hand.

[0064] Optionally, a pre-trained machine learning classifier suitable for pattern matching and / or instance matching can be provided. The pre-trained machine learning classifier can be trained using a sample set of tabular data files, which includes representative data files for parsing tabular data files, a set of successfully parsed tabular data files, and a set of unparsed tabular data files.

[0065] Before training a pre-trained machine learning classifier, the following steps can be performed. First, a representative tabular data file for a specific project has been uploaded, and several tables of interest within this file have been highlighted / selected by the user for the extraction process. This yields a template dictionary describing the location of each type of table of interest. The template dictionary can then be applied to all other files within the specific project. For each file, this process may be successful or unsuccessful, depending on data quality (i.e., cleanliness), effectively splitting the files into two sets. Furthermore, samples are selected from the failed set, and the reasons for their failure are investigated, such as tables being in the wrong location, columns being in the wrong order, etc. Based on the investigation, a list of unevenness is derived, which can be directly transformed into a sequence of operations required for successful data parsing.

[0066] This essentially provides all the inputs needed by a pre-trained machine learning classifier: a template dictionary and a set of successfully parsed files. Optionally, users can define their own similarity metrics for use on all target tables.

[0067] The pre-trained machine learning classifier can then be trained as follows: First, it constructs a similarity metric and prepares a logistic regression classifier. Then, it uses a template dictionary and applies it to successfully parsed files, from which correct training samples can be derived. From these training samples, it can also compute the pattern attributes of the target tables. In the same way, it derives incorrect training samples from files where the template failed. Furthermore, it uses successfully parsed files and slightly shifts the template to derive more invalid training samples. In this way, many so-called critical examples are obtained. For each target table, it then computes a similarity matrix based on the valid and invalid samples. The logistic regression classifier is then trained using this matrix along with vectors labeling rows as correct or incorrect. The weights learned by the classifier are used to update the similarity metric. Using the new similarity, for each target table, the centroid, i.e., the row with the highest inter-table similarity, is computed. The training phase then ends, and the pre-trained machine learning classifier is ready to parse the remainder of the files.

[0068] It's important to note that the pre-trained machine learning classifier determines the requirements for the transformation based solely on the output samples, since both the set of successfully parsed tabular data files and the set of failed parsings are derived from the sample set of the tabular data files. In other words, there are no input samples, and therefore no input-output example pairs. Thus, the user's workload remains constant regardless of the number of different layouts. Furthermore, the pre-trained machine learning classifier is pattern- and instance-based rather than location-based, making it applicable to other input tables with different layouts. In other words, it can be used when the tabular data files have structural biases.

[0069] In some embodiments, the user interface 12 may be configured to provide a report 42 that includes at least one of the following: an overview of extracted statistics, a response table showing the number of tables extracted from each table.

[0070] Figure 4 An example of a graphical report according to an embodiment of this disclosure is shown. In the illustrated example, the user interface displays a pie chart overview of the extracted statistics on the left. On the right side of the interface, a response table showing the number of tables extracted for each file is illustrated. These tables can be linked and migrated to a database by pressing simple buttons from the user interface. Furthermore, if needed, the user can download this collection as an Excel or CSV file at any time, for example, as input to other software or analysis pipelines.

[0071] In some implementations, the data extraction system 10 can be used in, for example... Figure 5The distributed database system 100 shown is implemented in, for example, but not limited to, distributed laboratory systems, distributed production systems, and distributed business systems. The example distributed database data retrieval system 100 shown includes multiple electronic communication devices 110, such as electronic communication devices 110a and 110b, and multiple data repositories 120, such as data repositories 120a, 120b, and 120c, a web server 130, and a network 110. For simplicity, only two electronic communication devices 110a and 110b and three data repositories are illustrated. However, the following discussion can be extended to a large number of electronic communication devices 110 and a large number of data repositories 120.

[0072] The illustrated electronic communication devices 110a and 110b can be desktop computers, laptops, mobile phones, smartphones, and / or PDAs. In some embodiments, the electronic communication devices 110a and 110b can also be referred to as clients. Each electronic communication device 110a and 110b can include user interfaces 12a and 12b configured to facilitate one or more users submitting one or more tabular data files 22, each tabular data file including at least one table 20 and a user-defined template 24 including at least one target table 26. The user interface 12 can be an interactive interface, including but not limited to a GUI, a character-based user interface, and a touchscreen interface. Optionally, the illustrated electronic communication devices 110a and 110b can include a storage device 50 for storing experimental data, production data, and / or business data.

[0073] Data repositories 120a, 120b, and 120c can be databases that receive data generated in production, business, and / or laboratory environments. For example, the three data repositories might collect experimental data from different laboratories. These laboratories may be located in the same physical location or in different cities, states, and / or countries, and they are interconnected by a network. In another example, the three data repositories may collect production data from different production sites, which may be located in the same physical location or distributed across different physical sites. The data repositories 120a, 120b, and 120c in the illustrated examples can be any type of database, including servers, databases, files, etc.

[0074] The web server 130 illustrated may be a server that provides network services to facilitate the management of data in multiple data repositories 120a, 120b, 120c and storage device 50. The web server 130 may include a data retrieval module 14, as described above and below. In some embodiments, the web server 130 may facilitate data management through user interaction, for example, via web pages, desktop applications, or mobile applications. Alternatively, the web server 130 illustrated may be replaced by another device (e.g., another electronic communication device) that provides any type of interface (e.g., command-line interface, graphical user interface). These interfaces, such as web pages, desktop applications, and mobile applications, may allow users to manage data via network 140 using electronic communication devices 110a, 110b. The web server 130 may also include an interface through which users can authenticate (by providing a username and password). For example, a user account may be used to authenticate system users to utilize the web server 130 to access some data repositories to submit one or more tabular data files stored thereon and to retrieve data from these tabular data files.

[0075] The illustrated network 140 communicatively couples multiple electronic communication devices 110a, 110b, multiple data repositories 120a, 120b, 120c, and a web server 130. In some embodiments, the network may be the Internet. Alternatively, network 140 may be any other type and number of networks. For example, network 140 may be implemented by several local area networks (LANs) connected to a wide area network (WAN). For example, electronic communication devices 110a and data repositories 120a may be associated with a first LAN of a first laboratory, while electronic communication devices 110b and data repositories 120b may be associated with a second LAN of a second laboratory. Web server 130 may be associated with a third LAN. The first, second, and third LANs may be connected to a WAN. Of course, network 140 may be implemented using any other configuration and topology, including any combination of wired networks, wireless networks, WANs, LANs, etc.

[0076] Figure 6A A flowchart illustrating a computer-implemented method 200 for extracting data from one or more tabular data files according to an embodiment of the present disclosure is shown.

[0077] In step 210, i.e., step a), one or more tabular data files are provided. Each tabular data file contains one or more tables. In some embodiments, the one or more tabular data files may be submitted by one or more users via a user interface. The user interface may be an interactive interface, including but not limited to a GUI, a character user interface, and a touchscreen interface. Users may evaluate the user interface via an electronic communication device, including but not limited to a desktop computer, a laptop computer, a mobile phone, or a smartphone. The one or more tabular data files may contain data of any type. Examples of data include, but are not limited to, data obtained from laboratory environments, production environments, and / or business-related environments. For example, the one or more tabular data files may include experimental data from a distributed laboratory system. Furthermore, the one or more tabular data files may have various file formats, such as, but not limited to, CSV format and data from… The document is in electronic form format.

[0078] Step b) includes steps b1) to b5 for extracting data from one or more tabular data files.

[0079] In step 220, i.e., step b1), a user-defined template including at least one target table is provided. The user can select the target table from existing table data files (i.e., historical data) in the database.

[0080] In step 230, i.e., step b2), target schema-level information for at least one target table is identified. Schema-level information relates to the inherent attributes of the table itself, and each candidate must populate these attributes, such as name, description, data type, relation type (part-of, is-a, etc.), constraints, and schema structure.

[0081] In step 240, i.e., step b3), at least one semantically matching candidate table is selected from one or more tabular data files based on the target pattern-level information. One or more methods can be used to identify the one or more candidate tables. For example, a language matcher can be used, which uses names and text including words or sentences to find semantically similar pattern elements. Name-based matching can match pattern elements with the same or similar names. A description-based matcher can be used, which determines the similarity between pattern elements based on annotations accompanying the pattern elements. A constraint-based matcher can be used, which determines similarity based on constraints defining, for example, data types and value ranges, uniqueness, optionality, relations, and cardinality. Typically, the pattern-level matcher 16 can find multiple matching candidates. For each candidate, the degree of similarity, limited by a normalized numerical value in the range of 0 to 1, can be estimated to identify suitable candidates to be output to the instance-level matcher.

[0082] In some embodiments, the pattern-level matcher is configured to transform at least one table in one or more submitted tabular data files into at least one transformed table to match a format defined by at least one target table and to select at least one semantically matching candidate table from the at least one transformed table. The transformation includes at least one of the following: extracting a sub-table from the at least one table, transposing the at least one table, permuting columns of the at least one table; and removing at least one column from the at least one table that contains mismatched pattern-level information.

[0083] In some embodiments, a sequence of operations is performed on at least one candidate table, the sequence of operations defining a tree structure with multiple edges and multiple nodes. Each edge corresponds to a basic transformation of a parent node having one or more free parameters, and each node corresponds to a transformed candidate table of its respective basic transformation. At least one transformed candidate table is selected at multiple leaf nodes of the tree structure, the similarity of the at least one transformed table to the at least one target table being higher than the similarity of the remaining transformed candidate tables to the at least one target table. An instance-level matcher is configured to extract data from the at least one selected transformed candidate table, the data having instance-level information that matches the target instance-level information.

[0084] In step 250, i.e., step b4), target instance-level information for at least one target table is identified. The instance-level information relates to the actual value of each cell.

[0085] In step 260, i.e., step b5), data is extracted from at least one candidate table of semantic matches based on target instance-level information. For example, a constraint-based matcher can be used, which determines the corresponding matching candidate based on constraint-based features, such as numerical range and average value or character pattern. For example, a record linking method can be used to identify records in the dataset that are linked to the same entity.

[0086] In some embodiments, a similarity metric is determined between instance-level information of at least one candidate table and target instance-level information of at least one target table, and data is extracted from at least one candidate table, wherein the determined similarity metric of the data exceeds a reference value. The similarity metric can be assigned in a cascading manner for each column or row of the target table: i) specifying a similarity metric for each column of the target table, ii) specifying a similarity metric for all columns of a specific type, and iii) setting all remaining columns to a default similarity, which is optionally user-defined.

[0087] This method learns only the column types (schema-level information) and values ​​(instance-level information), not their exact locations (not index-based), regardless of the table's position or whether it has changed. Therefore, due to formatting and / or experimental modifications, this method is applicable to data files from different laboratories, production units, and business departments.

[0088] Figure 6B A flowchart illustrating a computer-implemented method 200 for extracting data from one or more tabular data files according to another embodiment of the present disclosure is shown.

[0089] In step 210, i.e., step a), one or more tabular data files are provided. Each tabular data file contains one or more tables. In some embodiments, the one or more tabular data files may be submitted by one or more users via a user interface. The user interface may be an interactive interface, including but not limited to a GUI, a character user interface, and a touchscreen interface. Users may evaluate the user interface via an electronic communication device, including but not limited to a desktop computer, a laptop computer, a mobile phone, or a smartphone. The one or more tabular data files may contain data of any type. Examples of data include, but are not limited to, data obtained from laboratory environments, production environments, and / or business-related environments. For example, the one or more tabular data files may include experimental data from a distributed laboratory system. Furthermore, the one or more tabular data files may have various file formats, such as, but not limited to, CSV format and data from… Electronic form file formats, Open Document Format (ODF), Lotus Worksheet format Works electronic form format, Electronic form format, Abykus worksheet format, and Ability electronic form format.

[0090] In step 270, i.e., step c), the validator template is applied to at least one table in one or more tabular data files to validate the table format of the at least one table based on the template format of the validator template. Rules for tags, metadata, and / or data may include at least one of the following: i) the format of tags, numbers, data, and time; ii) the uniqueness of identifiers and attributes; iii) the completeness and uniqueness of metadata; and iv) the logical structure based on primary keys and foreign keys. Based on the analysis of one or more submitted tabular data files, the validator template may be derived from a user-defined schema.

[0091] It should be recognized that the above operations can be performed in any suitable order, such as consecutively, simultaneously, or a combination thereof, subject to a specific order required, for example, by the input / output relationship, where applicable.

[0092] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, characterized in that it is adapted to perform method steps of the method according to one of the foregoing embodiments on a suitable system.

[0093] Computer program elements can therefore be stored on a computer unit, which may also be part of embodiments of the present invention. This computing unit can be adapted to perform or guide the execution of the steps of the described methods. Furthermore, it can be adapted to operate components of the described apparatus. The computing unit can be adapted to automatically operate and / or execute user commands. The computer program can be loaded into the working memory of a data processor. The data processor can therefore be equipped to execute the methods of the present invention.

[0094] This exemplary embodiment of the invention covers both computer programs that use the invention from the outset and computer programs that convert existing programs into programs that use the invention through updates.

[0095] Furthermore, computer program elements may be able to provide all the necessary steps to implement the exemplary embodiments of the method described above.

[0096] According to another exemplary embodiment of the present invention, a computer-readable medium, such as a CD-ROM, is presented, wherein computer program elements are stored thereon, the computer program elements being described in the preceding section.

[0097] Computer programs may be stored and / or distributed on suitable media, such as optical or solid-state media provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0098] However, computer programs can also be presented via networks like the World Wide Web and downloaded from such networks to the working memory of a data processor. According to another exemplary embodiment of the invention, a medium is provided for making computer program elements available for download, these computer program elements being arranged to perform a method according to one of the foregoing embodiments of the invention.

[0099] It should be noted that embodiments of the present invention are described with reference to different subjects. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, those skilled in the art will understand from the above and below description that, unless otherwise notified, any combination of features relating to different subjects, except for any combination of features belonging to one class of subjects, is also considered to be disclosed with this application. However, all features can be combined together to provide more synergistic effects than the simple sum of these features.

[0100] While the invention has been detailed and described in the accompanying drawings and foregoing description, such description is to be considered illustrative or exemplary rather than restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments will be understood and implemented by those skilled in the art in practicing the claimed invention upon studying the drawings, the disclosure, and the dependent claims.

[0101] All definitions defined and used herein should be understood as control dictionary definitions, definitions referenced and entered into the file, and / or the general meaning of the defined terms.

[0102] The terms “a” and “an” as used in the specification and claims shall be understood as “at least one” unless expressly indicated to the contrary. In other words, the terms “a” and “an” do not exclude plural forms.

[0103] The phrase “and / or” as used in the specification and claims should be understood to mean “one or two” of the elements so combined, i.e., elements that are combined in some cases but not in others. Multiple elements listed with “and / or” should be interpreted in the same way, i.e., “one or more” of the elements connected in this way. In addition to the elements specifically identified by the “and / or” clause, other elements may optionally be present, whether related to or unrelated to those specifically identified.

[0104] As used herein in the specification and claims, the phrase "at least one" relating to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list, but not necessarily including at least one of each element specifically listed in the list, and does not exclude any combination of elements in the list. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements referred to by the phrase "at least one," whether related to or unrelated to those specifically identified elements.

[0105] In the claims and the foregoing description, all transitional phrases, such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “constituting,” etc., should be understood as open-ended, meaning including but not limited to. Only the transitional phrases “constituting of” and “constituting substantially of” are closed or semi-closed transitional phrases, respectively.

[0106] A single processor or other unit can perform the functions of several items recited in the claims. The fact that certain measures are referenced in mutually different dependent claims does not imply that a combination of these measures cannot be used advantageously. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A data extraction system (10) for extracting data from one or more tabular data files, the system comprising: User interface (12) adapted to facilitate one or more users to submit one or more tabular data files (22), each tabular data file including at least one table (24). The user interface is also configured to receive: i) A user-defined template (26) including at least one target table (28); The data extraction system further includes a data extraction module (14) with a pattern-level matcher (16) and an instance-level matcher (18). The pattern-level matcher is configured to identify target pattern-level information of the at least one target table, select at least one semantically matching candidate table from one or more submitted table data files based on the target pattern-level information (30), and transform at least one table in one or more submitted table data files into at least one transformed table to match the format defined by the at least one target table and select at least one semantically matching candidate table from the at least one transformed table: Perform a sequence of operations on at least one table in one or more submitted tabular data files, the sequence of operations defining a tree structure (34) with multiple edges (36) and multiple nodes (38), each edge corresponding to a basic transformation of a parent node using one or more free parameters, and each node corresponding to a transformation table of the corresponding basic transformation; At least one transformation table is selected at multiple leaf nodes (40) of the tree structure, wherein the similarity between the at least one transformation table and the at least one target table is higher than the similarity between the remaining transformation tables and the at least one target table; Select at least one semantically matching candidate table from at least one selected transformation table; and The instance-level matcher is configured to identify target instance-level information of the at least one target table and extract data from the at least one semantically matched candidate table based on the target instance-level information; or ii) Validator template; The data extraction system further includes a validator (20) configured to apply the validator template to at least one table in the one or more tabular data files to validate the table format of the at least one table based on rules defined by the validator template for tags, metadata and / or data.

2. The system according to claim 1, in, The instance-level matcher is configured to determine a similarity metric between the at least one semantically matched candidate table and the at least one target table, and to extract data from the at least one semantically matched candidate table whose similarity metric exceeds a given threshold.

3. The system according to claim 2, in, The similarity metric is assigned to each column of the target table in a cascading manner: i) Specify the similarity measure for each column of the target table; ii) Specify the similarity measure for all columns of a specific type; and iii) All remaining columns are set to the default similarity, which is optionally user-defined.

4. The system according to any one of claims 1-3, in, The transformation includes at least one of the following: - Extract a sub-table from at least one table; - Transpose at least one table; - Replace at least one column in a table; and - Remove at least one column or row from at least one table that contains mismatched schema-level information.

5. The system according to any one of claims 1-3, further comprising: - A pre-trained machine learning classifier suitable for performing pattern matching and / or instance matching; The pre-trained machine learning classifier is configured to be trained using a sample set of tabular data files, the sample set of tabular data files having representative data files for parsing the tabular data files, a set of successfully parsed tabular data files, and a set of failed tabular data files.

6. The system according to any one of claims 1-3, in, The user interface is configured to provide reports that include at least one of the following: - Extract an overview of the statistics; - A response table containing the number of tables extracted from each table data file, form, and / or table type; as well as - An interface that allows users to directly migrate extracted tables to the database.

7. The system according to claim 1, in, The rules for tags, metadata, and / or data include at least one of the following: i) the format of tags, numbers, data, and time; ii) the uniqueness of identifiers and attributes; iii) the completeness and integrity of metadata; and iv) the logical structure based on primary keys and foreign keys.

8. The system according to claim 1 or 7, in, The validator template can be exported from a user-defined schema based on the analysis of the one or more submitted tabular data files.

9. The system according to any one of claims 1-3, in, The one or more tabular data files have one or more formats, including at least one of the following: comma-separated value CSV format and formats from Microsoft... ® Excel ® The document is in the format of an electronic form file; and Optionally, one or more tabular data files to be verified by the verifier have one or more formats, including at least one of the following: Open Document Format (ODF), Lotus 1-2-3 ® Worksheet format, Microsoft ® Works electronic form format, VisiCalc ® Electronic form format, Abykus worksheet format, and Ability electronic form format.

10. A distributed database system, comprising: - One or more data repositories; as well as - A data extraction system according to any one of the preceding claims; The one or more data repositories are configured to store tabular data files from multiple sources, each tabular data file including at least one table; and The data extraction system is configured to extract data from the tabular data file based on a user-defined template that includes at least one target table.

11. A computer-implemented method (200) for extracting data from one or more tabular data files, comprising: a) Provide (210) one or more tabular data files, each tabular data file including one or more tables; as well as Perform step b) or step c); Step b) includes: b1) Provide (220) a user-defined template that includes at least one target table; b2) Identify target pattern-level information of at least one target table of (230); b3) Select (240) at least one semantically matching candidate table from one or more submitted tabular data files based on the target pattern-level information; The following operations transform at least one table in one or more submitted tabular data files into at least one transformed table to match the format defined by at least one target table, and select at least one semantically matching candidate table from the at least one transformed table: Perform a sequence of operations on at least one table in one or more submitted tabular data files, the sequence of operations defining a tree structure (34) with multiple edges (36) and multiple nodes (38), each edge corresponding to a basic transformation of a parent node using one or more free parameters, and each node corresponding to a transformation table of the corresponding basic transformation; At least one transformation table is selected at multiple leaf nodes (40) of the tree structure, wherein the similarity between the at least one transformation table and the at least one target table is higher than the similarity between the remaining transformation tables and the at least one target table; Select at least one semantically matching candidate table from at least one selected transformation table; b4) Identify the target instance-level information of at least one target table of (250); and b5) Extract (260) data from the at least one semantically matched candidate table based on the target instance-level information; and Step c) includes applying (270) a validator template to at least one table in the one or more table data files to validate the table format of the at least one table based on the template format of the validator template.

12. A computer program product comprising computer program instructions for instructing a system according to any one of claims 1 to 10, the computer program instructions being adapted, when executed by a processing unit, to perform the steps of the method according to claim 11.

13. A computer-readable medium having stored thereon computer program instructions for instructing a system according to any one of claims 1 to 10, wherein the computer program instructions, when executed by a computer device, cause the computer device to perform the method according to claim 11.

Images