Configurable data extraction form and associated method

Abstract

A method and apparatus for generating a configurable data extraction form is disclosed. The apparatus receives identifiers for measurement variables and classes, and a display format defining the ordering of the classes on a user interface. A data extraction form is generated, and its layout can be dynamically re-rendered by modifying the class ordering, such as by swapping rows and columns, without altering the underlying populated data. The system can also automatically screen new articles for duplicates using a similarity score. For unique articles, the configurable form is automatically generated and displayed. The form is structured by mapping dimensions to rows and columns, with headings providing contextual guidance for entering measure values. Instructional prompts, such as questions, can be automatically generated by programmatically combining these headings to further guide the user during data entry.

Description

CONFIGURABLE DATA EXTRACTION FORM AND ASSOCIATED METHODTechnical Field

[0001] The present disclosure relates to the field of research articles review, and in particular to the gathering of data such as measures and categories of measures from multiple research articles.Background

[0002] Reviewing research articles, e.g., medical research articles, requires the reviewer to extract data from the articles, analyze the data, and tabulate the data for presentation. Each research article can present its respective data in a format different from that of other articles, which makes it challenging for the reviewer to compare data from different articles and analyze the data.

[0003] When reviewing articles, the reviewer will typically create a spreadsheet with columns and rows into which measures (i.e., values of measurement variables) may be entered, and fields into which categories of data (qualitative or categorical data) may be entered. As an example, articles on type 1 diabetes may include, as measurement variables, the number of participants in the study, the incidence rate of type 1 diabetes among the participants, the confidence interval of the incidence rate, etc. As categories of data, the articles may include the sex of the participant, the study type and the World Health Organization region of study. Some of the articles may include all the measurement variables, others may include only some of them. Similarly, some of the articles may include all the categories of data, others may include only some of them.

[0004] To capture all the measures and all the categories of data of all the articles being reviewed, the reviewer typically constructs a spreadsheet that has cells into which the measures (the values of all the measurement variables values) of all the articles can be entered and fields (groups of cells) into which the measures of respective categories of data can be entered. As the reviewer typically reviews the articles in sequence, the spreadsheet constructed for reviewing the first article will typically need to be modified for subsequent articles through the addition of rows, columns, cells, and fields.

[0005] As will be appreciated by the skilled worker, reviewing tens or hundreds of articles can lead to a large and cumbersome spreadsheet from which intelligible trends or patterns may be difficultto extract. Therefore, improvements in the way data is extracted from medical study articles are desirable.Summary

[0006] The present disclosure provides a method and system configured to generate a configurable data extraction form that allows a user to select predefined measurement variables or generate new measurement variables; and select categories of data fields or generate new categories of data fields specifically for each article being reviewed. The data model underlying the method and system allows storage of all the data collected from all the articles being reviewed and the presentation of some data or all the data from different articles in a configurable presentation format such as, for example, tables, plots, graphs, lists, etc.

[0007] The present disclosure provides a computer-implemented method and apparatus for generating a highly configurable data extraction form, designed to solve the technical problem of efficiently and consistently capturing data from research articles that present information in varied and heterogeneous formats. The invention is centered on a system that allows a user to define the layout of a data extraction form based on a specified ordering of data classes. This allows the user interface to be dynamically reconfigured to mirror the structure of a source document, streamlining the data entry process. The system is built upon a specific data model of "dimensions" (categorical data) and "measures" (numerical data), which a user can arrange to programmatically generate a structured, table-based form where the row and column headings provide clear contextual guidance.

[0008] A further aspect of the invention involves the automation of the initial review workflow. The system can automatically screen new articles against a database of existing articles to identify unique, non-duplicate entries using a similarity score. Upon identifying a unique article, the system seamlessly integrates the next step by automatically generating and displaying the appropriate configurable data extraction form alongside the article, creating an efficient and streamlined process for the reviewer. The invention also contemplates additional features, such as the use of artificial intelligence models for advanced article screening and the generation of discrete instructional prompts, to further guide the user during data entry.

[0009] Advantageously, the present disclosure allows a supervisor of an article review project to specify to the reviewers the data points the reviewers are to collect when reviewing articles. Thereviewers can in turn configure (generate) the data extraction form according to the supervisor’s directives. Once collected, the data can be rearranged and analyzed regardless of the format (or manner) it was collected in. The present disclosure also allows easy comparison of data collected across articles (studies), even when the data is captured heterogeneously by the reviewers.

[0010] According to one aspect of the present disclosure, there is provided a computer- implemented method, comprising: receiving identifiers for at least one measurement variable and at least one class; receiving a display format defining a relative ordering of the one or more classes; generating a data extraction form in accordance with the display format; populating data extraction fields in response to user data extraction; and displaying the data extraction form. Advantageously, this method provides user interface flexibility in that the user is not confined to a rigid, predefined form. Instead, they can dynamically adapt the data entry interface to match the specific layout of any given source article, which significantly speeds up the data extraction process and reduces the cognitive load on the reviewer, minimizing errors.

[0011] In an embodiment, generating the data extraction form further comprises mapping a first class to define rows of a table and a second class to define columns of the table, the method further comprising: receiving a subsequent user input that modifies the relative ordering of the classes; and in response, dynamically re-rendering the populated data extraction form in a new row and column configuration without altering the underlying populated data.

[0012] In another embodiment, the display format is a first display format, and the method further comprises receiving a second display format and displaying the data extraction form in accordance with the second display format.

[0013] According to another aspect of the present disclosure, there is provided a computer- implemented method, comprising: obtaining and analyzing a new article; comparing the new article against previous articles to obtain a similarity score; responsive to the similarity score failing to satisfy a predetermined criteria, identifying the new article as unique; and in response, generating and displaying a configurable data extraction form alongside the unique article by mapping classes to rows and columns of a table. Advantageously, this aspect improves workflow automation and efficiency. By programmatically identifying unique articles and automatically presenting them within the configurable data extraction environment, the system eliminates severalmanual steps in the systematic review process. This integration prevents duplicate work and creates a seamless transition from article screening to data extraction, increasing user throughput.

[0014] In an embodiment, the comparison is effected by string matching based on an exact or partial match filter to produce the similarity score, which is provided to a user to signal a duplicate when the score is high.

[0015] In an embodiment, the method further comprises screening articles using artificial intelligence based on user-defined inclusion and exclusion criteria.

[0016] In an embodiment, the artificial intelligence comprises a support vector machine (SVM) model trained by articles that have been previously included and excluded.

[0017] In an embodiment, the method further comprises: generating time frequency-inverse document frequency (TF-IDF) values from the articles; creating a binary classifier using the TF-IDF values; and using the classifier to predict the status of unscreened articles and generate a confidence value.

[0018] In an embodiment, the method further comprises providingthe prediction and confidence value to a user and receiving an indication from the user to sort, filter, or adjudicate the prediction.

[0019] According to a further aspect of the present disclosure, there is provided a computer- implemented method for generating a structured data extraction form, comprising: defining a plurality of data elements, each comprising at least one dimension and at least one measure; receiving a user selection to arrange a first dimension to define rows of a table and a second dimension to define columns; and in response, automatically generating and displaying the data extraction form, wherein the row and column headings provide contextual guidance.Advantageously, this provides data integrity and clarity in that the method enforces a consistent and structured data collection process. By using a defined data model of dimensions and measures to generate a clear, table-based interface, the system ensures that all captured data is uniform and analysis-ready on the backend, while the contextual headings provide unambiguous guidance to the user on the front-end, improving the quality and reliability of the extracted data.

[0020] In an embodiment, the contextual guidance is provided by automatically generating a discrete instructional prompt for each data entry cell by programmatically combining the corresponding row and column headings.

[0021] In an embodiment, programmatically combining the headings further comprises including an identifier for the associated measure to form a complete instructional prompt.

[0022] In an embodiment, the identifier for the measure includes a unit type that is included in the discrete instructional prompt.

[0023] In an embodiment, the instructional prompt is a question or an instruction.

[0024] According to another aspect of the present disclosure, there is provided an apparatus comprising a processor and a memory storing instructions that, when executed, configure the processor to perform the methods summarized above.

[0025] According to a further aspect of the present disclosure, there is provided a tangible computer-readable medium having instructions recorded thereon to be performed by a processor to carry out the methods of any one of the methods summarized above.Brief Description of the Figures

[0026] Further features and advantages of the present invention will become apparent from the following detailed description, taken in combination with the appended drawings, in which:

[0027] FIG. 1 shows a class diagram of a data model, in accordance with an embodiment of the present disclosure.

[0028] FIG. 2 shows a user interface (Ul) configured for a user to select existing data elements to be extracted when reviewing an article or for the user to create such data elements, in accordance with an embodiment of the present disclosure.

[0029] FIG. 3 shows a Ul that includes pre-existing data elements that relate to psychological tests, in accordance with an embodiment of the present disclosure.

[0030] FIG. 4 shows a Ul configured to allow for a data extraction form to be selected or generated, in accordance with an embodiment of the present disclosure.

[0031] FIG. 5 shows a Ul configured to receive text for questions that a supervisor of an article review project would like answered, in accordance with an embodiment of the present disclosure.

[0032] FIG. 6 shows a Ul with a form pre-viewer portion and a workflow conditions portions, in accordance with an embodiment of the present disclosure.

[0033] FIG. 7 shows a Ul with specific workflow conditions, in accordance with an embodiment of the present disclosure.

[0034] FIG. 8 shows an in-use Ul that includes a data extraction form, in accordance with an embodiment of the present disclosure.Detailed Description

[0035] FIG. 1 shows a class diagram 50 of an embodiment of a data model in accordance with the present disclosure. Any other suitable data model is considered to be within the scope of the present disclosure. The data model in this embodiment includes a Project 52, which has one or more categories of data (dimensions) 54 and one or more measures 56. Each category of data 52 has a name and one or more dimensional values 58. As an example, a category of data may be a region of the world and be called “WHO Region of Study” (WHO: World Health Organization) and the dimensional values associated with this category of data may include, e.g., Afghanistan, Albania, etc.

[0036] With respect to the one or more measures 56, they may each include a unit type, an aggregation type, and a name. For example, a measure may be the number of participants in a study and may be named (called) “Number of Participants”. The aggregation type in this case is a count aggregation. The validation of the data occurs when the number of participants is equal or greater than zero. The unit type in this example is unitless (the number of participants is a number without any units).

[0037] The project 52 also includes one or more references 60, each of which includes a data extraction collection (DECollection) 62 that has a project identifier (projectID), a reference ID (refID), and user interface metadata (uiMetaData), which can include the categories selected by the reviewer, the order of the selected categories, etc.) The DECollection 62 also include data extraction cells (DECell) 64, which are also included in the reference 60. The DECell 64 includes a collection ID (collectionlD) and an intersection ID (intersectionlD). The intersectionlD relates to the intersection of categories and to the data associated with that intersection.

[0038] The DECell 64 includes a DE intersection (DEIntersection) 66 that includes a cell ID (celUD), a dimension ID (dimensionlD), and a dimensional value ID (dimensionalvaluelD). The DECell 64 also includes a measure value (Measurevalue) 68 that has a value and a measuredID.

[0039] The present disclosure may provide a user interface (Ul) that allows the user to create data elements or select pre-defined data elements for categories of data and for measures. The data elements can be made available for any project associated with an account. FIGs. 2 and 3 show examples of how such data elements can be created.

[0040] The present disclosure provides a flexible data extraction form that is enabled by the data model's architecture, as shown in the embodiment of FIG. 1 . The DECell 64 entity acts as a central hub for each unique data point. It links a specific Measurevalue 68, which contains the raw numerical data (e.g., a mean of 6.94), to a unique combination of categorical values through the DEIntersection 66 table. Each record in DEIntersection 66 associates the DECell 64 with a specific Dimension 54 and a specific DimensionalValue 58 (e.g., Dimension: 'Psyc Tests', DimensionalValue: 'DASS Anxiety'). By storing data in this decoupled manner, the system separates the underlying data from its presentation. This allows the user interface to dynamically rearrange the display format, for example, by swapping the rows and columns, without altering the stored data itself, as the relationships between measures and their corresponding dimensional values are maintained independently of the visual layout.

[0041] FIG. 2 shows an example of a Ul where the user can create or select the data elements to be considered in reviewing articles. The data elements can be provided by selection of pre-existing data element options or by creating new data elements. When a user wishes to create a new data element, the Ul presents several options. These options include a Dimension (closed), which is a categorical data element used for questions with a predefined set of answers, such as a multiplechoice list or dropdown menu. Another option is a Dimensions (Open), which is a categorical data element used to capture unstructured or free-text input. The user can also create a Measure for numerical data points and a Date and Time element for calendar information.

[0042] FIG. 3 shows an example of a Ul that includes pre-existing data elements that relate to psychological tests (Psyc Tests). The dimension is named “Psyc Tests”, has the short name “Psyc Tests), and is configured to be added on the fly, meaning that when the reviewer reviews a first article that does not have a psychological test dimension, then the reviewer can add that dimension to the data extraction form. The Ul of FIG. 3 also shows an interactive table that includes the categories available for the Psyc Test data element. The categories include Beck’s Depression Inventory (BDI), Depression Anxiety Stress Scales (DASS) Depression, and DASS Anxiety, which are marked as selected, and Cognitive Reactivity, which is marked as not selected. The Ul shown atFIG. 3 also includes question inputs boxes (“A Question”, “Another Question”) in which questions to be considered / answered by the reviewer can be added by the supervisor of the reviewer or by any other suitable member of the article review team.

[0043] FIG. 4 shows an example of a Ul that allows for a data extraction form (or data capture form) to be generated or for the data extraction form to be selected from pre-existing data extraction forms such as, for example, a “Title Screening Form” configured for the reviewer to extract data from the title of the article, an “Abstract Screening Form” configured for the reviewer to extract data from the abstract of the article, a general data extraction (DE) form “DE General Form”, configured for the reviewer to extract data from the article, or specialized DE forms configured for the reviewer to extract data from specialized articles that may have specialized data categories and measures. The Ul of FIG. 4 also allows for the selection or creation of data elements that are to be part of the data extraction form. In the example of FIG. 4, there are no pre-existing data form elements. To provide guidance to a user who populates the form, the system can automatically generate a discrete instructional prompt for a data entry cell. In the context of the present disclosure, an instructional prompt is a user interface element that provides guidance or direction to a user on how to correctly enter data or complete a task. The purpose of an instructional prompt is to ensure that information is entered accurately and consistently by clearly indicating the required data for a specific field. In the context of the present disclosrue application, several elements function as instructional prompts. As an example, explicit questions: A supervisor can create direct questions, such as "What tests were used in the study?", that are associated with specific data entry fields to guide the reviewer. As another example, structural headings: In a table-based form, the row and column headings (e.g., "Probiotic" or "DASS depression") serve as prompts, instructing the user on what data to enter at their intersection. As a further example, generated prompts: The system can programmatically (automatically) combine row and column headings to automatically generate a discrete instructional prompt for a specific data cell. This prompt is generated by programmatically combiningthe row and column headings correspondingto that cell. The prompt may be further enhanced by including an identifier for the measure and its associated unit type. Furthermore, when data elements are grouped into a table on the form, embodiments of the system of the present disclosure can automatically generate template questions based on the measures and dimensions selected, providing guidance to the user who populates the form.

[0044] FIG. 5 shows an example of Ul configured to receive the text of the question or questions that the supervisor of the article review would like answered. In the example of FIG. 5, there is only one question, which reads “What tests were used in the study?” and is set as being a mandatory question, meaning that every article reviewed is expected to have an answer to this question. The Ul is set such that the question in the example allows multiple answers. Further, the Ul of FIG. 5 indicates that the default answer to the question is selected and that the default answer is “Randomized Control Trial”. The Ul of this example is also configured such that any question entered at the Ul is added to the existing questions, if any, and that questions can be added on the fly.

[0045] FIG. 6 shows a Ul with a “form previewer” portion and “workflow conditions” portions. The Ul has been populated with the form element “Count of Participants”, which is associated with the question “How many participants in the group?”; the form element (category) “Psyc Tests”, which is associated with the question “What tests were used in the study?” and with the selectable answers (categories) “BDI”, “DASS Depression”, “DASS Anxiety”, and “DASS Stress”; and the form element (category) “Country”, which is associated with the selectable countries “Afghanistan”, “Albania”, etc.

[0046] FIG. 7 shows an example of workflow conditions that can be programmed by a review supervisor to automatically route articles to different review forms based on extracted data. The logic can be built using multiple conditions and operators. For instance, as shown, a rule can be created where if the extracted data indicates that the 'BDI' test is selected OR the 'DASS Anxiety' test is selected, THEN the article is sent to the "DE 'Specialist 1 ' FORM". The workflow engine can also process multiple distinct rule sets in sequence. As another example, a second rule may send the article to the "DE 'Specialist 2' FORM" under the same conditions. Importantly, the workflow also supports an ELSE condition, which directs any articles that do not meet the preceding criteria to a default next step, such as, for example, the "Risk of Bias Form". This allows for the creation of comprehensive and automated article processing pipelines.

[0047] FIG. 8 shows an in-use example of Ul with a data extraction form, when reviewing an article (“Attachment #1 ”). In this example, the article in question is: Bahia Chahwan et al. (2019), Gut feelings: A randomised, triple-blind, placebo-controlled trial of probiotics for depressive symptoms, J. of Affective Disorders, Vol. 253, pp. 317-326.

[0048] In the example of FIG. 8, the data extraction form lists, in the category of psychological tests, “BDI”, “DASS depression”, “DASS anxiety”, “DASS stress”, “Cognitive Reactivity”, and “BAI” (Beck Anxiety Inventory). The data extraction form also lists, in the category of timepoints, “Pre” (pre-intervention), “Post” (post intervention), and “Follow-up” (for depressed subjects, one month follow-up). The data extraction form also lists, in the category of interventions, “Probiotic” and “Placebo”. As for the measurements, the data extraction form includes cells to be filled by the reviewer of the article, when reviewingTable 2 of the article. For DASS anxiety, post, probiotic, the reviewer would enter 6.94 with a standard deviation of 9.25. In this example, the data extraction form has the same format as Table 2 of the article, however, this need not be the case. If a reviewer comes across another article where the data is presented as in the table shown here:Tablethe data extraction table can be easily modified to match the appearance of the table above by rearranging the order of the rows in the settings of the form, i.e., by moving the “Psyc Tests” element 100 below the “Timepoint” element 102. That is, the relative placement of one category with respect to another, on the U I, may change the format of the data extraction form.

[0049] In some implementations, the data extraction cells can be configured to automatically change the units of the data being entered. For example, if a diabetes related article shows blood glucose readings milligram per deciliter (mg / dL) units, the data extraction cells could be made to automatically convert the blood glucose entry to millimoles per liter (mmol / L) by multiplying the blood glucose value measured in mg / dL by 0.0555. For example, a value of 126 mg / dL would automatically be converted to 7.0 mmol / L.

[0050] In some embodiments, the system includes features to assist users in managing and screening the collection of articles within the database. Before a new article is added, it can be compared against the plurality of previous articles already stored to detect potential duplicates.This process involves analyzing the new article to extract values from predetermined fields, such as the title, authors, and abstract. For each previous article, the system compares these string values against the corresponding values of the new article to calculate a similarity score. The matching can be performed using an exact match filter or a more flexible partial match filter. If the resulting similarity score satisfies a predetermined threshold, the system flags the new article as a potential duplicate and notifies the user. Conversely, when the similarity score fails to satisfy the predetermined criteria, the system identifies the new article as unique and, in response automatically generates and displays the configurable data extraction form alongside the unique article, ready for data entry.

[0051] The system may further employ artificial intelligence to accelerate the article screening process based on user-defined inclusion and exclusion criteria. In one implementation, a Support Vector Machine (SVM) model is trained using a set of articles that have already been manually included or excluded by the user. As the user screens more articles, the model continuously learns the criteria. This trained binary classifier can then predict the status (i.e., include or exclude) of the remaining unscreened articles.

[0052] To enhance the predictive model, textual features from the articles, such as the abstract, can be vectorized using a Term Frequency-Inverse Document Frequency (TF-IDF) algorithm. The resulting TF-IDF values for key terms are used by the binary classifier to make its prediction and to generate a confidence value forthat prediction. The system presents both the prediction and the confidence value to the user, who can then sort or filter the remaining articles based on this information and choose to agree or disagree with the automated prediction, thereby providing further training data for the model.

[0053] As another example, for a project related to the review of type 1 diabetes articles, the categories and their possible values may include:1. Sex of Participants a. Male b. Female c. Both2. Study Type a. Longitudinal b. Prospective Cohort c. Randomized ControlTrial d. Other3. WHO Region of Study a. Western Pacific b. Americas c. European d. South-East Asia e. Eastern Mediterranean4. Study Setting a. Rural b. Urban c. Mixed5. Median Year of Data Collection a. Before 1985 b. 1985 - 1990 c. 1990 - 1995 d. 1995 - 2000 e. 2000 - 2005 f. 2005 - 2010 g. After 20106. Weight Assessment Method a. Measured Directly b. Self Reported7. Diabetes Ascertainment (Multiple Selection) a. Blood Tests b. Medical Records c. Self-Reported8. Weight Groups by BM I a. Underweight (BMI of <18.5) b. Normal Weight (BMI of 18.4-22.9 for Asians and 18.5-24.9 for non-Asians) c. Overweight (GMI of >23 for Asians, >25 for Non-Asians)The measurement variables (measures) may include, for example:1. Number of Participants a. Aggregation Type = Count b. Validation = > 0 c. Unit Type = Unitless2. Incidence Rate a. Aggregation Type = Incidence Rate b. Validation = > 0 c. Unit Type = Unitless3. Confidence Interval (Cl) for Incidence Rate a. Aggregation Type = Confidence Interval b. Validation = > 0 c. Unit Type = Unitless4. Person Years a. Aggregation Type = Countb. Validation = > 0 c. Unit Type = Time5. Cases a. Aggregation Type = Count b. Validation = > 0 c. Unit Type = Unitless6. Follow Up Duration (Years) a. Aggregation Type = Count b. Validation = > 0 c. Unit Type = Time i. Group when reporting into <8 and >8 to use as Closed Dimension

[0054] Embodiments of the present disclosure may allow a user to shape the data extraction form for an article being reviewed to the form the article uses to present the data.

[0055] Each action or operation of the method described herein may be executed on any computing device, such as a personal computer, server, PDA, or the like and pursuant to one or more, or a part of one or more, program elements, modules or objects generated from any programming language, such as C++, Java, or the like. In addition, each operation, or a file or object or the like implementing each said operation, may be executed by special purpose hardware or a circuit module designed for that purpose.

[0056] Through the descriptions of the preceding embodiments, the present invention may be implemented by using hardware only or by using software and a necessary universal hardware platform. Based on such understandings, the technical solution of the present invention may be embodied in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), USB flash disk, or a removable hard disk. The software product may include a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided in the embodiments of the present invention. For example, such an execution may correspond to a simulation of the logical operations as described herein. The software product may additionally or alternatively include number of instructions that enable a computer device to execute operations forconfiguringor programminga digital logic apparatus in accordance with embodiments of the present invention.

[0057] The word “a” or “an” when used in conjunction with the term “comprising” or “including” in the claims and / or the specification may mean “one”, but it is also consistent with the meaning of“one or more”, “at least one”, and “one or more than one” unless the content clearly dictates otherwise. Similarly, the word “another” may mean at least a second or more unless the content clearly dictates otherwise.

[0058] The terms “coupled”, “coupling” or “connected” as used herein can have several different meanings depending on the context in which these terms are used. For example, as used herein, the terms coupled, coupling, or connected can indicate that two elements or devices are directly connected to one another or connected to one another through one or more intermediate elements or devices via an electronic element depending on the particular context. The term “and / or” herein when used in association with a list of items means anyone or more of the items comprising that list.

[0059] Although a combination of features is shown in the illustrated embodiments, not all of them need to be combined to realize the benefits of various embodiments of this disclosure. In other words, a system or method designed according to an embodiment of this disclosure will not necessarily include all features shown in any one of the figures or all portions schematically shown in the Figures. Moreover, selected features of one example embodiment may be combined with selected features of other example embodiments.

[0060] Although the present invention has been described with reference to specific features and embodiments thereof, it is evidentthatvarious modifications and combinations can be made thereto without departingfrom the invention. The specification and drawings are, accordingly, to be regarded simply as an illustration of the invention as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present invention.

Claims

CLAIMS:

1. A computer-implemented method, comprising: receiving, for at least one measurement variable, a respective identifier of the at least one measurement variable; receiving, for at least one class of the at least one measurement variable, a respective identifier of the at least one class; receiving a display format defining a relative ordering of the one or more classes on a user interface, the display format for displaying one or more measurement variables of the at least one measurement variable and one or more class of the at least one class; generating a data extraction form in accordance with the display format, the data extraction form having data extraction fields associated with the at least one measurement variables and the at least one class; populating the data extraction fields in response to data extraction effected by a user, to obtain a populated data extraction form; and displaying the data extraction form in accordance with the display format.

2. The method of claim 1 , wherein generating the data extraction form further comprises mapping a first class of the at least one class to define rows of a table and a second class of the at least one class to define columns of the table, the method further comprising: receiving a subsequent user input that modifies the relative ordering of the first class and the second class; and in response to the subsequent user input, dynamically re-rendering the populated data extraction form to display the data extraction fields in a new row and column configuration corresponding to the modified relative ordering, wherein the re-rendering occurs without altering the underlying populated data.

3. The method of claim 1 , wherein: the display format is a first display format, the method further comprising: receiving a second display format for displaying the one or more measurement variables of the at least one measurement variable and the one or more class of the at least one class; anddisplaying the data extraction form in accordance with the second display format.

4. A computer-implemented method, comprising: obtaining a new article to determine if the new article is to be stored into a database, the database storing a plurality of previous articles indexed in accordance with predetermined fields; analyzing the new article to obtain values of the predetermined fields of the new article, the values being string values; for each previous article of the plurality of previous articles, comparing the values of the predetermined fields of the new article against values of the predetermined fields of the previous article, to obtain a similarity score; and responsive to determining that the similarity score fails to satisfy a predetermined criteria, identifying the new article as a unique article; and in response to identifying the new article as a unique article, generating and displaying a configurable data extraction form alongside the unique article, wherein generating the data extraction form comprises mapping a first class to define rows of a table and a second class to define columns of the table, the table including data extraction fields for receiving user input correspondingto the unique article.

5. The method of claim 4 wherein the match is effected by string matching based on an exact match filter or a partial match filter, the result of the string matching using the exact match filter, and the partial match filter is related to a similarity score provided to the user and the user signals that the second article is a duplicate of the first article when the similarity score is high.

6. The method of claim 4 further comprising a screening of articles included in a collection of articles in the database, the screening using artificial intelligence using inclusion and exclusion criteria provided by the user.

7. The method of claim 6 further comprising a support vector machine (SVM) model trained by articles that have been included and excluded.

8. The method of claim 4 further comprising time frequency-inverse document frequency (TF- IDF) values of one or more abstract words, the TF-IDF values used to create a binary classifier and the binary classifier used to predict the status of remaining unscreened articles of the collection of articles (included / excluded) and also a confidence value representing an indication that the remaining unscreened articles should be removed.

9. The method of claim 8 wherein the prediction and confidence value are provided to the user who can indicate their desire to sort or filter the remaining unscreened articles and also agree or disagree with the prediction.

10. A computer-implemented method for generating a structured data extraction form, the method comprising: defining, in a tangible computer-readable memory, a plurality of data elements, each data element of the plurality of data elements comprising at least one dimension representing a categorical value and at least one measure representing a numerical value to be extracted; and receiving a user selection to arrange at least a first dimension to define rows of a table and a second dimension to define columns of the table; and in response to the user selection, automatically generating and displaying the data extraction form, the form comprising a plurality of data entry cells at the intersections of the rows and columns, wherein the row and column headings provide contextual guidance for a user to input a value for the at least one measure into a corresponding data entry cell.11 . The method of claim 10, wherein the contextual guidance is provided by automatically generating a discrete instructional prompt for each data entry cell, the prompt being generated by programmatically combining the row heading of the first dimension and the column heading of the second dimension corresponding to that data entry cell.

12. The method of claim 11 , wherein programmatically combining the row heading and the column heading further comprises including an identifier for the at least one measure associated with the data entry cell, thereby forming a complete instructional prompt for the user.

13. The method of claim 11 , wherein the identifier for the at least one measure includes a unit type, and wherein the unit type is included in the discrete instructional prompt to guide the user in entering data in a correct format.

14. The method of claim 11 , wherein the instructional prompt is a question or an instruction.

15. An apparatus, comprising: a processor; and a memory communicatively coupled to the processor and storing instructions thereon, wherein the instructions, when executed by the processor, configure the processor to: receive, for at least one measurement variable, a respective identifier of the at least one measurement variable; receive, for at least one class of the at least one measurement variable, a respective identifier of the at least one class; receive a display format defining a relative ordering of the one or more classes on a user interface, the display format for displaying one or more measurement variables of the at least one measurement variable and one or more class of the at least one class; generate a data extraction form in accordance with the display format, the data extraction form having data extraction fields associated with the at least one measurement variables and the at least one class; populate the data extraction fields in response to data extraction effected by a user, to obtain a populated data extraction form; and display the data extraction form in accordance with the display format.

16. The apparatus of claim 15, wherein the processor, when generating the data extraction form, is further configured to map a first class of the at least one class to define rows of a table and a second class of the at least one class to define columns of the table; and wherein the processor is further configured to: receive a subsequent user input that modifies the relative ordering of the first class and the second class; andin response to the subsequent user input, dynamically re-render the populated data extraction form to display the data extraction fields in a new row and column configuration corresponding to the modified relative ordering, wherein the re-rendering occurs without altering the underlying populated data.

17. The apparatus of claim 15, wherein the display format is a first display format, and wherein the processor is further configured to: receive a second display format for displaying the one or more measurement variables of the at least one measurement variable and the one or more class of the at least one class; and display the data extraction form in accordance with the second display format.

18. An apparatus, comprising: a processor; and a memory communicatively coupled to the processor and storing instructions thereon, wherein the instructions, when executed by the processor, configure the processor to: obtain a new article to determine if the new article is to be stored into a database, the database storing a plurality of previous articles indexed in accordance with predetermined fields; analyze the new article to obtain values of the predetermined fields of the new article, the values being string values; for each previous article of the plurality of previous articles, compare the values of the predetermined fields of the new article against values of the predetermined fields of the previous article to obtain a similarity score; responsive to determining that the similarity score fails to satisfy a predetermined criteria, identify the new article as a unique article; and in response to identifying the new article as a unique article, generate and display a configurable data extraction form alongside the unique article, wherein generating the data extraction form comprises mapping a first class to define rows of a table and a second class to define columns of the table, the table including data extraction fields for receiving user input corresponding to the unique article.

19. The apparatus of claim 18, wherein the processor is configured to effect the comparison via string matching using an exact match filter or a partial match filter, and wherein the processor is further configured to present the similarity score to a user and receive a signal indicating that the new article is a duplicate when the similarity score is high.

20. The apparatus of claim 18, wherein the processor is further configured to screen articles included in a collection of articles in the database using artificial intelligence based on inclusion and exclusion criteria provided by a user.21 . The apparatus of claim 20, wherein the artificial intelligence comprises a support vector machine (SVM) model trained by articles that have been included and excluded.

22. The apparatus of claim 1 , wherein the processor is further configured to: generate time frequency-inverse document frequency (TF-IDF) values of one or more abstract words from the articles; create a binary classifier using the TF-IDF values; and use the binary classifier to predict a status of remaining unscreened articles and to generate a confidence value.

23. The apparatus of claim 5, wherein the processor is further configured to: provide the prediction and the confidence value to a user; and receive an indication from the user to sort or filter the remaining unscreened articles or to agree or disagree with the prediction.

24. An apparatus, comprising: a processor; and a memory communicatively coupled to the processor and storing instructions thereon, wherein the instructions, when executed by the processor, configure the processor to: define, in the memory, a plurality of data elements, wherein each data element comprises at least one dimension representing a categorical value and at least one measure representing a numerical value to be extracted;receive a user selection to arrange at least a first dimension to define rows of a table and a second dimension to define columns of the table; and in response to the user selection, automatically generate and display a data extraction form, the form comprising a plurality of data entry cells at the intersections of the rows and columns, wherein the row and column headings provide contextual guidance for a user to input a value for the at least one measure into a corresponding data entry cell.

25. The apparatus of claim 24, wherein the contextual guidance is provided by the processor automatically generating a discrete instructional prompt for each data entry cell by programmatically combining the row heading of the first dimension and the column heading of the second dimension corresponding to that data entry cell.

26. The apparatus of claim 25, wherein the processor, when programmatically combining the row heading and the column heading, is further configured to include an identifier for the at least one measure associated with the data entry cell, thereby forming a complete instructional prompt for the user.

27. The apparatus of claim 25, wherein the identifier for the at least one measure includes a unit type, and wherein the processor is further configured to include the unit type in the discrete instructional prompt to guide the user in entering data in a correct format.

28. A tangible computer-readable medium having instructions recorded thereon to be performed by a processor to carry out the method of any one of claims 1 to 14.

Images