A system and method for inserting data into a structured database based on a graphical representation of a data table.

Abstract

1. A system and method for populating a structured database, the system and method comprising: accessing a graphical representation of a data table including one or more cells arranged in rows and columns; providing the graphical representation as input to a neural network model; running the neural network model to identify a location of a first content object within the graphical representation; identifying a location of a first cell based on the location of the first content object; determining that the first cell belongs to a first row and a first column based on the location of the first cell and the first content object in relation to a plurality of content objects; associating the first content object with one or more classification identifiers; and populating the structured database with the first content object and the one or more classification identifiers.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 874,830, filed on Jul. 16, 2019, entitled “Systems and Methods for Populating a Structured Database Based on an Image Representation of a Data Table”, the entire content of which is incorporated herein by reference.

[0002] This application generally relates to databases, and more specifically, to techniques for inserting data into a structured database based on an image representation of a data table.

Background Art

[0003] Database technology enables vast amounts of data to be digitally stored and accessed in an efficient manner. For example, many emerging “big data” applications are enabled by the development of database technology. Databases can be stored locally within a data center and / or within the cloud. Databases can also be distributed across multiple facilities.

[0004] Databases can be structured in various ways. For example, relational databases model data as a set of tables where each of the data is arranged in rows and columns. Query languages can be used to programmatically access data from a database and manipulate the data stored within the database.

[0005] However, inserting information into a database and keeping that information accurate and up - to - date can be a difficult task. Therefore, it is desirable to develop improved techniques for inserting data into a database, including automated techniques suitable for large - scale collection and storage of information within the database. [Overview of the project]

[0006] A system and method for inserting data into a structured database based on an image representation of a data table, according to embodiments of the present disclosure, includes accessing the image representation of a data table by one or more computer processors, wherein the data table includes one or more cells arranged in one or more rows and one or more columns, and the one or more cells include a first cell belonging to at least one first row and at least one first column, in which a first content object is inserted; providing the image representation as input to a neural network model trained to identify the locations of content objects in the image representation by one or more computer processors; executing the neural network model to identify the locations of the first content objects in the image representation by one or more computer processors; identifying the locations of the first cells based on the locations of the first content objects by one or more computer processors; The data insertion includes determining that a first cell belongs to at least one first row and a first column based on the location of the first cell and one or more of the first content objects in relation to a plurality of content objects associated with one or more rows and one or more columns; associating the first content object with one or more classification identifiers; and inserting information associated with the first content object and one or more classification identifiers into a structured database based on the determination by one or more computer processors that the first cell belongs to at least one first row and at least one first column, wherein the structured database includes at least one data table row associated with at least one first row and at least one data table column associated with at least one first column.

[0007] In some embodiments, the system and method may also include creating at least one second column and at least one second row in a structured database based on determining that a first cell does not belong to at least one first row and at least one first column. In some embodiments, accessing an image representation includes receiving a digital document over a computer network by one or more computer processors, wherein the digital document includes a data table; rendering the digital document as a digital image by one or more computer processors; and locating the image representation of the data table in the rendered digital image by one or more computer processors. In other embodiments, the location of a first content object includes a first region corresponding to at least a portion of the first content object, and identifying the location of a first cell based on the location of the first content object includes extending the first region in at least one direction, determining that the extended first region includes a graphical marker marking one or more row boundaries and column boundaries, and, in response to determining that the extended first region includes a graphical marker, identifying the extended first region as corresponding to the location of a first cell.

[0008] In some embodiments, determining that an extended first region contains a graphical marker includes: identifying a plurality of pixel locations corresponding to the edge of the extended first region; determining, for each pixel location within the plurality of pixel locations, whether the pixel location is associated with one or more changes in color and intensity along at least one extension direction that exceeds a first predetermined threshold; determining that the count of the plurality of pixel locations associated with the changes in color or intensity exceeds a second predetermined threshold; and determining that the extended first region contains a graphical marker in response to determining that the number of the plurality of pixel locations exceeds the second predetermined threshold. In other embodiments, the location of a first cell includes a row span along the row axis and a column span along the column axis, and determining that the first cell belongs to at least one first row and at least one first column based on the location of the first cell includes sorting at least a subset of one or more cells in a data table based on multiple locations of multiple cells, and recursively performing an operation to identify one or more second cells belonging to the first row, starting from a selected cell among the subset of one or more cells, wherein the operation includes determining at least one other cell having a row span that overlaps with the row span of the selected cell, and at least one other This includes identifying the cell closest to the selected cell among the cells, identifying the closest cell as belonging to at least one first row, selecting the closest cell as the next selected cell, identifying the header row among one or more rows of the data table based on one or more header content objects in which data has been inserted into one or more header cells of the header row, determining that the column span of the first cell overlaps with the column span of the first header cell among one or more header cells, and identifying the first cell as belonging to a first column, wherein the first column is associated with the first header cell.

[0009] In some embodiments, identifying a header row from one or more rows in a data table includes generating one or more text representations corresponding to one or more header content objects, querying a header dictionary for each of the one or more text representations, yielding a score vector containing one or more confidence scores corresponding to the one or more text representations, and each confidence score being determined based on the strength of the query, determining a row score based on the score vector, and selecting a header row based on the row score. In other embodiments, determining a row score based on the score vector includes calculating an aggregate metric based on the score vector and one or more of the one or more confidence scores. In yet another embodiment, selecting a header row includes comparing the row score to at least one secondary row score associated with one or more rows in the data table, and selecting a header row based on the relative values ​​of the row score and the at least one secondary row score.

[0010] In some embodiments, the system and method include: obtaining a list of excluded header content objects that are not eligible to be part of a header row using one or more computer processors; determining, using one or more computer processors, whether one or more header content objects into which data has been inserted in one or more header cells of a header row match any of the excluded header content objects; and, if one or more header content objects are on the list of excluded header content objects, identifying a replacement header row among one or more rows of a data table based on the one or more header content objects into which data has been inserted in one or more header cells of a header row using one or more computer processors. In other embodiments, the first content object includes a graphical sequence object, and data insertion into a structured database includes extracting sequence information from the graphical sequence object, with the information associated with the first content object including sequence information. [Brief explanation of the drawing]

[0011] [Figure 1] This is a simplified diagram of a system for inserting data into a structured database based on an image representation of a data table, according to several embodiments. [Figure 2] This is a simplified diagram of a data table in several embodiments. [Figure 3] This is a simplified diagram of a method for inserting data into a structured database based on an image representation of a data table, according to several embodiments. [Figure 4] This is a simplified diagram of a method for accessing an image representation of a data table, according to several embodiments. [Figure 5] This is a simplified diagram of a method for identifying the location of a cell based on the location of a content object, according to several embodiments. [Figure 6] This is a simplified diagram of a method for determining whether a region contains a graphical marker, according to several embodiments. [Figure 7] This is a simplified diagram of a method for determining, based on the cell's location, that a cell belongs to at least one row and at least one column, according to several embodiments. [Figure 8] This is a simplified diagram illustrating a method for identifying a header row from one or more rows in a data table, according to several embodiments. [Figure 9A] This is a simplified diagram of a drug information table according to several embodiments. [Figure 9B] This is a simplified diagram of a drug information table according to several embodiments. [Figure 9C] This is a simplified diagram of a drug information table according to several embodiments. [Figure 9D] This is a simplified diagram of a drug information table according to several embodiments. [Figure 9E]A simplified diagram of a pharmaceutical information table according to some embodiments. [Figure 9G] A simplified diagram of a pharmaceutical information table according to some embodiments. [Figure 9G] A simplified diagram of a pharmaceutical information table according to some embodiments. [Figure 10A] A simplified diagram of a pharmaceutical information table in which logical groups of text are automatically identified according to some embodiments. [Figure 10B] A simplified diagram of a pharmaceutical information table in which logical groups of text are automatically identified according to some embodiments. [Figure 10C] A simplified diagram of a pharmaceutical information table in which logical groups of text are automatically identified according to some embodiments. [Figure 10D] A simplified diagram of a pharmaceutical information table in which logical groups of text are automatically identified according to some embodiments. [Figure 11A] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments. [Figure 11B] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments. [Figure 11C] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments. [Figure 11D] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments. [Figure 11E] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments. [Figure 11F] A simplified diagram of a pharmaceutical information table in which cells are identified as belonging to specific rows and columns according to some embodiments.

[0012] The various objectives, features, and advantages of the disclosed subject matter can be more fully clarified by referring to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings in which like reference numerals identify like elements.

Best Mode for Carrying Out the Invention

[0013] Due to the various formats and structures of tables, extracting information from table data can be a difficult task. The rows and columns of a data table can have variable widths, heights, and spacing. A data table may or may not have row or column markers to identify the boundaries between adjacent rows or columns. Some data tables contain merged cells. Additionally, a data table can include a complex background or cell discoloration scheme.

[0014] For example, a biopharmaceutical company may provide a web page or downloadable report containing pharmaceutical pipeline information. This information is often presented in tabular form. For example, a pharmaceutical pipeline information table can include various information about products in development, such as drug name, target, mechanism of action, disease, and current stage of development. The stage of development can be presented graphically, for example, using progress bars of different shapes, sizes, and colors. A particular progress bar can span multiple columns, and even if the column markers appear to divide the cell containing the progress bar into multiple columns, the cell should be treated as a merged cell.

[0015] The present disclosure describes systems and methods for extracting information from data tables, such as those described above, and storing them in a structured database for subsequent retrieval and analysis.

[0016] Figure 1 is a simplified diagram of a system 100 for inserting data into a structured database based on an image representation of a data table, according to several embodiments. The system 100 includes a plurality of devices 101-109 that are connected communicably via a network 110. Devices 101-109 generally include computer devices or systems such as personal computers, mobile devices, and servers. The network 110 may include one or more local area networks (LANs), wide area networks (WANs), wired networks, wireless networks, and the Internet. Exemplaryly, devices 101-109 may communicate via the network 110 using the TCP / IP protocol or other preferred network protocol.

[0017] One or more of devices 101-109 may store and / or access digital documents 121-129 via the network 110. For example, as shown in Figure 1, devices 101, 102, and 109 store digital documents 121, 122, and 129, respectively, and device 103 accesses digital documents 121-129 via the network 110. Digital documents 121-129 may include web pages, digital files, digital images (including one or more frames of a video or animation), etc. Exemplary examples, digital documents 121-129 may be formatted as HTML / CSS documents, PDF documents, word processor documents (e.g., Word documents), text documents, slideshow presentations (e.g., PowerPoint presentations), image files (e.g., JPEG, PNG, or TIFF images), etc. For efficient storage and / or transmission over network 110, documents 121-129 may be compressed before or during transmission over network 110. Security measures such as encryption, authentication (including multi-factor authentication), SSL, HTTPS, and other security technologies may also be applied.

[0018] According to some embodiments, device 103 may access one or more of the digital documents 121-129 by downloading them from devices 101, 102, and 109. Furthermore, one or more of devices 101, 102, or 109 may upload the digital documents 121-129 to device 103. The digital documents 121-129 may be updated at various times. Therefore, device 103 may access the digital documents 121-129 multiple times at various intervals (e.g., periodically) to obtain the latest copy.

[0019] At least one of the digital documents 121-129 may contain one or more data tables 131-139. For example, data tables 131-139 may be embedded within digital documents 121-129 or linked from within digital documents 121-129. Data tables 131-139 may be stored in various formats, such as image format, text format (e.g., CSV or TSV file), or markup language format (e.g., XML or HTML / CSS).

[0020] As shown in Figure 1, device 103 includes a processor 140 (e.g., one or more hardware processors) coupled to memory 150 (e.g., one or more non-temporary memories). Memory 150 stores instructions and / or data corresponding to a processing pipeline 160 and a neural network model 170 (or multiple neural network models). When executed by processor 140, the processing pipeline 160 inserts data into the database 180 based on the image representations of data tables 131-139. Since digital documents 121-129 can generally be stored and accessed in various formats, the processing pipeline 160 may convert digital documents 121-129 and / or data tables 131-139 into image representations in preparation for processing. This preliminary conversion step allows the processing pipeline 160 to process data tables received in HTML / CSS and PDF formats, for example, using the same technique.

[0021] Database 180 may be configured as a structured database with content organized according to schemes or other logical relationships. For example, database 180 may be a relational database. Although database 180 is illustrated as being directly linked to device 103, it should be understood that various other configurations are possible. For example, database 180 may be stored in memory 103 or accessed via network 110.

[0022] While the processing pipeline 160 is running, the processor 140 runs the neural network model 170. The neural network model 170 is trained to make predictions based on input data. The neural network model 170 includes a configuration 172 that defines the multiple layers of the neural network model 170 and the relationships between the layers. Exemplary examples of layers include input layers, output layers, convolutional layers, tightly connected layers, merge layers, and so on. In some embodiments, the neural network model 170 may be configured as a deep neural network having at least one hidden layer between the input and output layers. The connections between layers may include feedforward connections or recurrent connections.

[0023] One or more layers of the neural network model 170 are associated with trained model parameters 174. The trained model parameters 174 are a set of parameters (e.g., weights and bias parameters of artificial neurons) that are learned according to a machine learning process. During the machine learning process, labeled training data is provided as input to the neural network model 170, and the values ​​of the trained model parameters 174 are repeatedly adjusted until the predictions generated by the neural network 170 match the corresponding labels with a desired level of accuracy.

[0024] For improved performance, processor 140 may run the neural network model 170 using a graphical processing unit, a tensor processing unit, an application-specific integrated circuit, and the like.

[0025] Figure 2 is a simplified diagram of data table 200 in several embodiments. In some embodiments consistent with Figure 1, data table 200 may generally correspond to at least one of data tables 131 to 139.

[0026] The data table 220 contains one or more cells 231-239 located in one or more rows 241-249 and one or more columns 251-259. Generally, each cell belongs to at least one row and at least one column. Furthermore, one or more of the cells 231-239 may correspond to merged cells that occupy multiple rows, multiple columns, or both. For example, as illustrated in Figure 2, cell 235 corresponds to merged cells spanning columns 252-259.

[0027] Content objects 261-269 will be inserted into one or more of the cells 231-239. Content objects 261-269 can contain various types of content, such as text, graphics, equations, animated content, or a combination of these.

[0028] According to some embodiments, one or more of the content objects 261-269 may include a graphical sequence object. For example, as shown in Figure 2, cell 235 is data-inserted with content object 269 which includes a graphical sequence object 270. The graphical sequence object 270 represents sequence information, such as timing or phase information. For example, the graphical sequence object 270 may represent the development stage of a project, the clinical trial phase of a drug, etc. In some embodiments, the graphical sequence object 270 may illustrate the sequence information using progress bars, and the length of the progress bars (e.g., the number of columns the progress bars span) conveys the sequence information. Generally, graphical sequence objects such as the graphical sequence object 270 may be provided in a wide variety of shapes, sizes, colors, textures, patterns, etc.

[0029] One or more of rows 241-249 may be designated as the header row of the data table 200. For example, as shown in Figure 2, the top row 241 is designated as the header row. The contents of the header row include information that describes the contents of other rows, such as text labels contained within the cells of the columns below the individual cells of the header row. For example, cells 232 and 233 in header row 241 are data-inserted with content objects 261 and 262, which contain header content objects 282 and 284, respectively. Header content object 282 contains information that describes the contents of other rows in column 252, and header content object 284 contains information that describes the contents of other rows in column 259.

[0030] In some embodiments, adjacent columns or rows of the data table 200 may be separated using graphical markers, such as graphical column markers 292 or graphical row markers 294. While graphical column markers 292 and graphical row markers 294 are illustrated as solid lines in Figure 2, many alternatives are possible. For example, graphical markers may include lines of various styles (e.g., dashed, dotted, double lines), background colors, or style transitions (e.g., adjacent rows or columns may be separated by alternating between light and dark background colors, or between different textures). As will be recognized by those skilled in the art, graphical markers can be applied in a wide variety of ways depending on the style and content of the data table 200. Graphical markers may be included for some rows and / or columns, while they may be omitted for others.

[0031] Figure 3 is a simplified diagram of a method 300 for inserting data into a structured database based on an image representation of a data table, according to several embodiments. According to several embodiments consistent with Figures 1 and 2, the method 300 may be carried out by a computer processor, such as a processor 140, based on instructions and / or data stored in memory, such as memory 150.

[0032] In process 301, an image representation of a data table, such as data table 200, is accessed. The image representation includes pixel data representing the data table. An exemplary embodiment of accessing the image representation of a data table is described below with reference to Figure 4.

[0033] In process 302, a neural network model, such as neural network model 170, is accessed. The neural network model is trained to identify the location of content objects in the image representation. Exemplaryly, content objects may correspond to logical groups of text, such as text boxes. Thus, the neural network model can be trained to identify logical groups of text in the image representation. For example, the neural network model may include a text detector that detects words in the image representation, and a heuristic approach may be used to identify logical groups of detected words.

[0034] In some embodiments, a neural network model can be trained to directly identify logical groups of text. An example of a neural network model capable of identifying logical groups of text in this way is the YOLOv3 neural network, which is described in Joseph Redmon and Ali Farhadi, YOLOv3: An Incremental Improvement, Technical report, 2018, which is incorporated herein by reference in its entirety.

[0035] In some embodiments, a neural network model may be trained using transfer learning to identify one or more types of content objects that are expected to be found in the data table. For example, the neural network may be trained to identify (1) logical groups of text within a cell (e.g., text boxes) and (2) graphical sequence objects (e.g., progress or phase bars). Subsequent processes of Method 300 may be performed for each type of content object identified by the neural network model.

[0036] In process 303, the image representation is provided as input to the neural network model. Various preprocessing steps may be performed to prepare the image representation of the neural network model. These preprocessing steps may include cropping and / or padding the image representation to fit a given aspect ratio, scaling the dimensions of the image representation to fit a given size, normalizing the color or intensity of pixels in the image representation, and reducing the number of color channels in the image representation (e.g., converting the image representation from color to grayscale).

[0037] In process 304, a neural network model is executed to identify the location of a first content object within the image representation. The first content object may include a logical group of text, a graphical sequence object, and the like. According to some embodiments, the neural network model may be executed using dedicated computing hardware such as a graphics processing unit (GPU) or an application-specific integrated circuit (ASIC). The location of the first content object may include the coordinates of a point associated with the first content object (e.g., the center position of the first content object), the horizontal and vertical spans of the first content object, a bounding rectangle (or other preferred shape) surrounding the first content object, and the like.

[0038] More generally, running a neural network model can identify the locations of multiple content objects within an image representation. Process 304 and subsequent processes of Method 300 are described with reference to a first content object among multiple content objects, but these processes can be repeated for each of the multiple identified content objects.

[0039] In process 305, the location of a first cell is identified based on the location of a first content object. The first cell corresponds to a cell in a data table into which the first content object is inserted. The first cell corresponds to a region of the image representation that is generally greater than or equal to the first content object, since the first content object is contained within the first cell. Thus, identifying the location of a first cell can be achieved by extending the region corresponding to the first content object until the extended region reaches the boundary associated with the first cell. Exemplary embodiments of the method for identifying the location of a first cell in this way are described below with reference to Figures 5 and 6. According to some embodiments, process 305 may be repeated for each of the multiple content objects identified in process 304, resulting in the locations of multiple corresponding cells in the data table. In this regard, each of the multiple cells may be associated with a different content object and may have a different location.

[0040] In process 306, it is determined that the first cell belongs to at least one first row and at least one first column, based on the location of the first cell. Generally, cells in a data table belong to a single row and a single column. However, the first cell may correspond to a merged cell, in which case the first cell may span multiple rows, multiple columns, or both. Exemplary embodiments of a method for determining that the first cell belongs to at least one first row and at least one first column are described below with reference to Figure 7.

[0041] In process 307, data is inserted into a structured database, such as database 180, with information associated with a first content object based on the determination that a first cell belongs to a first row and a first column. Inserting data into a structured database may include extracting information based on the first content object. For example, if the first content object contains a logical group of text, inserting data into a structured database may include converting the logical group of text from an image representation into a sequence of digital characters.

[0042] When the first content object includes a graphical sequence object, inserting data into a structured database may include extracting sequence information from the graphical sequence object. For example, when the graphical sequence object includes progress or phase bars, the sequence information may be determined based on the length of the progress or phase bars, or the number of rows or columns that the progress or phase bars span. In some embodiments, the length of the progress or phase bars is first aligned to the columns or rows of the table before the sequence information is determined. In some embodiments, determining the length of the progress or phase bars may include distinguishing between the filled and unfilled portions of the bar and identifying the length of the filled portion. In scenarios where the progress or phase bars span a portion of a column or row, the percentage of overlap may be determined. For example, if a phase bar spans 60% of the column corresponding to Phase II, it may be determined that Phase II is 60% complete.

[0043] According to several embodiments, one or more processes of Method 300 may be repeated until data is inserted into the structured database with information associated with each content object in the data table. Once the data is inserted, various types of analysis or visualization can then be performed based on the information stored in the structured database. As an exemplary example, in several embodiments, semantic analysis may be performed based on the information stored in the structured database using, for example, the technique described in U.S. Patent No. 10,360,507, filed September 22, 2017, entitled "Systems, Methods, and Computer Readable Media for Visualization of Sematic Information and Inference of Temporal Signals Indicating Salient Associations Between Life Science Entities," which is incorporated herein by reference in its entirety.

[0044] Figure 4 is a simplified diagram of method 400 for accessing an image representation of a data table, according to several embodiments. According to several embodiments consistent with Figures 1 to 3, method 400 can be used to implement process 301 of method 300.

[0045] In process 401, digital documents such as digital documents 121-129 are received via a computer network such as network 110. Digital documents can be sent and received in various formats. For example, digital documents may include HTML / CSS documents, image files (e.g., JPEG, PNG, or TIFF images), PDF documents, text or word processing documents, slideshow presentations, spreadsheets, etc.

[0046] In process 402, the digital document is rendered as a digital image. For example, rendering a digital document may involve converting the digital document into an array of pixel values ​​that can be used for further processing (and optionally displayed on a display screen). The rendering engine may be selected to render the digital document into a uniform image format based on the format in which the digital document is received. For example, when the digital document contains an HTML / CSS document, a web browser may be selected to render the document. Similarly, when the digital document contains a PDF document, a PDF viewer may be selected to render the document. In either case, the digital document may be rendered into a uniform digital image format, regardless of the format of the received digital document. This provides flexibility for handling a wide variety of received digital document types. In some embodiments, metadata associated with the received digital document (e.g., metadata from the PDF file describing the contents of data tables contained within the PDF file) may be removed from the rendered digital image or otherwise omitted.

[0047] In process 403, the image representation of the data table is located within a rendered digital image. Those skilled in the art will understand that a wide variety of object detection techniques can be used to locate the image representation of the data table within the digital image. According to some embodiments, a second neural network model may be trained to detect and localize the data table within the digital image. This second neural network model may then be run using the rendered digital image as input to predict the location of the image representation of the data table. In exemplary embodiments, the neural network model may correspond to the SSD512 neural network model, which is trained using transfer learning to detect and localize the image representation of the data table. The SSD512 neural network model is described in more detail in Wei Liu et al., SSD: Single Shot MultiBox Detector, European Conference on Computer Vision, 2016, which is incorporated herein by reference in its entirety.

[0048] In some embodiments, method 400 may be performed multiple times to update the data table over time. For example, if the data table contains phase or progress information that changes or progresses over time, method 400 may be performed periodically to track the indicated phase or progress. Then, a method such as method 300 may be performed each time the data table is updated to insert data into a structured database based on the updated contents of the data table.

[0049] Figure 5 is a simplified diagram of method 500 for identifying cell locations based on the location of content objects, according to several embodiments. According to several embodiments consistent with Figures 1 to 3, method 500 may be used to implement process 305 of method 300.

[0050] In process 501, a first region corresponding to at least a portion of a first content object is expanded in at least one direction. For example, the first region may correspond to a bounding rectangle surrounding the first content object, e.g., a box around a logical group of text. In some embodiments, the edges of the bounding rectangle may be aligned parallel to the expected direction of rows and / or columns of a data table. For example, if rows and columns correspond to the horizontal and vertical axes of an image representation, respectively, the edges of the bounding rectangle may similarly be aligned with the horizontal and vertical axes of the image representation. However, other shapes (e.g., non-rectangular shapes) and / or orientations of the first region are equally applicable to the systems and methods described herein. When the first region corresponds to a bounding rectangle, in order to expand this first region, at least one of the four edges of the bounding rectangle may be shifted outward from the center of the bounding rectangle. The expansion may occur in steps of a predetermined size, e.g., increments of one pixel.

[0051] Process 502 determines whether the extended first region contains graphical markers that demarcate row or column boundaries. Graphical markers may include lines that demarcate row or column boundaries. Lines may generally have any preferred style (e.g., solid, dashed, patterned, colored, etc.). Graphical markers may also include transitions that convey row or column boundaries, such as changes in background color or texture. More generally, graphical markers may include any preferred type of discontinuity that conveys that a boundary between rows or columns exists at a given location in the image. Various image processing techniques may be used to detect whether such graphical markers are contained within the extended first region. Exemplary embodiments of methods for determining whether the extended first region contains graphical markers are described below with reference to Figure 6.

[0052] Process 503 determines whether the expanded first region overlaps with the second region, which corresponds to at least a portion of the second content object. For example, the expanded first region may overlap with the second region when the first and second content objects are in adjacent rows or columns and there is no graphical marker between the adjacent rows or columns. In these scenarios, the first and second regions may expand during process 501 and continue to grow until they overlap each other. Therefore, comparing the first region with other identified regions in the image that contain the second region may be performed to detect cells that do not have graphical markers defining their boundaries.

[0053] In process 504, in response to (a) process 502 determining that the extended first region contains a graphical marker, or (b) process 503 determining that the extended first region overlaps with the second region, the extended first region is identified as corresponding to the location of the first cell. In some embodiments, one or more processes of method 500 may be repeated until each cell boundary (e.g., two-row boundary and two-column boundary) is determined in a similar manner.

[0054] Figure 6 is a simplified diagram of method 600 for determining whether a region contains a graphical marker, according to several embodiments. According to several embodiments consistent with Figures 1 to 5, method 600 may be used to implement process 502 of method 500.

[0055] In process 601, multiple pixel locations corresponding to the edges of the extended first region are identified. For example, when the extended first region corresponds to an N×M rectangle, the multiple pixel locations may include N pixels along the right or left edge of the extended first region, or M pixels along the upper or lower edge of the extended first region. According to some embodiments, the extended first region generally corresponds to the extended first region associated with process 501.

[0056] In process 602, for each pixel position, it is determined whether the pixel position is associated with a change in color or intensity along at least one expansion direction that exceeds a first predetermined threshold. For example, if multiple pixel positions correspond to the left end of an N×M boundary rectangle, each pixel can be compared with its adjacent pixel to the right. During the comparison, the difference between the pixel and the adjacent pixel (e.g., intensity difference, color difference, etc.) can be calculated. The difference may be an absolute difference, a relative difference, etc. The difference is then compared with the first predetermined threshold. The first predetermined threshold is preferably set to a value high enough to avoid false positives (e.g., incorrectly detecting row or column boundaries based on a gradual background gradient) and low enough to detect fine-grained types of graphical markers (e.g., small but abrupt alternating background color transitions between rows).

[0057] In process 603, it is determined whether the count of multiple pixel locations associated with a change in color or intensity, as determined in process 602, exceeds a second predetermined threshold. The count may correspond to an absolute count of pixels or a relative count (e.g., a percentage of the total number of pixels). Some types of graphical markers may be continuous (e.g., solid lines), in which case each of the multiple pixels is likely to be included in the count. However, other types of graphical markers may be discontinuous (e.g., dashed lines), in which case fewer pixels than all of the multiple pixels are likely to be included in the count. Therefore, the second predetermined threshold is preferably set to a value low enough to detect discontinuous types of graphical markers without introducing false positives.

[0058] In process 604, in response to determining that the count of multiple pixel positions exceeds a second predetermined threshold, it is determined that the extended first region contains a graphical marker. In making this determination, a method such as method 500 may proceed to identify the region as corresponding to a cell location, as described in process 504.

[0059] Figure 7 is a simplified diagram of a method 700 for determining, based on the cell's location, that a cell belongs to at least one row and at least one column, according to several embodiments. According to several embodiments consistent with Figures 1 to 3, method 700 may be used to implement process 306 of method 300.

[0060] In process 701, multiple cells in the data table (for example, multiple cells identified in process 305 of method 300) are sorted based on their identified locations. For example, multiple cells may be sorted sequentially along the column axis (for example, left to right or right to left) and the row axis (for example, top to bottom or bottom to top).

[0061] In process 702, one or more cells from a plurality of sorted cells belonging to at least one row are recursively identified. According to some embodiments, recursively identifying one or more cells belonging to at least one row may include recursively performing the following actions, starting with a first selected cell: (1) determining a set of cells whose row span overlaps the row span of the currently selected cell; (2) identifying the nearest cell in the set of cells; (3) identifying the nearest cell as belonging to at least one row; and (4) selecting the nearest cell as the next selected cell. The row span corresponds to a range of positions occupied by cells along the row axis (e.g., the vertical axis of the data table). These actions may be performed from left to right (identifying the cell closest to the right of the selected cell) and from right to left (identifying the cell closest to the left of the selected cell) until each cell of at least one row has been identified.

[0062] In process 703, a header row is identified based on one or more header content objects into which data is inserted in one or more header cells of the header row. A header content object generally describes the contents of the corresponding column, for example, by providing an indicator or label. Thus, a given column in a data table can be identified based on the corresponding header content object for that column. The corresponding individual cells within that column will then have similar content objects that share common characteristics or data types identified by the header content object for that column. An exemplary embodiment of a method for identifying a header row is described below with reference to Figure 8.

[0063] Process 704 determines whether the column span of a first cell overlaps with the column span of at least one first header cell between one or more header cells. The column span corresponds to the range of positions occupied by a cell along the column axis (e.g., the horizontal axis of the data table). When the column spans of different cells overlap, the two cells are likely to belong to the same column. In the case of merged cells, the column span of a first cell may overlap with multiple header cells.

[0064] In process 705, a first cell is identified as belonging to at least one first column, and at least one first column is associated with at least one first header cell. However, if there is no header row, or if there is no header cell for at least one first column, an alternative approach may be used. For example, at least one first column may be assigned to a default header, such as featureless placeholder header text, when no header cell exists.

[0065] Furthermore, identifiers or labels for at least one first column may be predicted and assigned based on semantic analysis of the contents of at least one first column. For example, when there are no header cells for at least one first column, the text contained within the column's cells may be extracted and parsed using an entity extraction engine to determine the type of entity contained within the column. In some embodiments, the entity extraction engine may associate the text contained within the cells based on the type of entity, without having header cells to provide context within the data structure. For example, among other things, the technique for identifying entity types disclosed in U.S. Patent No. 10,360,507 may be used for this purpose. Identifiers or labels for the column may then be assigned based on the type of entity within the column, and the structured database may be populated based on the identifiers or labels. The identifiers or labels may be in the form of qualitative labels, such as noting the type of drug, target, disease, mechanism of action, or phase of the test. Various preprocessing steps may be applied to the text in preparation for sending it to the entity extraction engine. For example, text may be sent to a spell correction engine to correct misspelled or irregularly spelled text in preparation for an entity extraction engine. Exemplarily, in the context of pharmaceutical or biomedical applications, the spell correction engine may include a biomedical spell correction engine, and the entity extraction engine may include a biomedical entity extraction engine. Exemplary examples of entity types recognized by biomedical entity extraction include, but are not limited to, genes, drugs, tissues, diseases, organic chemicals, companies, diagnostic procedures, and physiological functions.

[0066] Figure 8 is a simplified diagram of method 800 for identifying a header row from one or more rows of a data table, according to several embodiments. According to several embodiments consistent with Figures 1 to 7, method 800 may be used to implement process 702 of method 700.

[0067] Process 801 generates one or more text representations corresponding to one or more header content objects. These one or more text representations may include a set of digital characters. According to some embodiments, optical character recognition (OCR) may be used to generate one or more text representations based on an image representation of a data table.

[0068] In process 802, each of one or more textual representations is queried against a header dictionary. This yields a score vector containing one or more confidence scores corresponding to one or more textual representations. Each confidence score is based on the strength of the query. For example, each confidence score may be determined based on the Levenshtein distance, which provides a mechanism to account for errors and uncertainties (e.g., OCR errors) in previous process steps. Exemplarily, in the context of a drug information table, the header dictionary contains entities corresponding to headers expected to be included in such a table, such as drug name, disease / target, mechanism of action, and phase. In some embodiments, the header dictionary is created by an expert ("SME") to manually identify common entries expected in the data system. The header dictionary may be updated over time to consider new common entity types, either manually or using an automated text recognition system. As discussed above, in some embodiments, semantic analysis may be carried out based on information stored in a structured database, for example, using the technique described in U.S. Patent No. 10,360,507, filed September 22, 2017, entitled "Systems, Methods, and Computer Readable Media for Visualization of Sematic Information and Inference of Temporal Signals Indicating Salient Associations Between Life Science Entities," which is incorporated herein by reference in its entirety.

[0069] In process 803, the row score is determined based on the score vector. The row score is an aggregate metric based on one or more confidence scores that make up the score vector. For example, the row score may be calculated as the sum of the square roots of the score vector.

[0070] In process 804, header rows are selected based on row scores. For example, a header row's row score may be compared to the row scores of other candidate rows in the data table. Candidate rows may include other rows with row scores greater than a given threshold (e.g., zero). Rows containing certain types of content objects may be excluded from the set of candidate rows. For example, a row containing a graphical sequence object (e.g., a phase bar) may be ineligible to be selected as a header row. Rows may also be excluded as eligible header rows based on whether or not they contain certain content objects, which may be defined by the SME using a list of excluded content objects that are ineligible to have data inserted into the header row. The header row may then be selected in response to having the highest row score among the candidate rows.

[0071] Figures 9A–9G are simplified diagrams of drug information tables 900a–g according to several embodiments. In some embodiments consistent with Figures 1–8, drug information tables 900a–g may correspond to data tables 131–139. As illustrated in Figures 9A–9G, the visual and substantial differences shown between drug information tables 900a–g reflect real-world differences in how drug information may be delivered. Despite this wide variability in how information is delivered, systems 100 and methods 200–800 may be configured to automatically parse and interpret drug information tables 900a–g and, based on the information in the tables, insert data into a structured database such as database 180. In some embodiments, the information obtained from the drug production information tables 900a-g may include columns or rows for the destination website or URL, the drug or development program name, the target population or disease of the study, the mechanism of action of the test drug, the phase information values ​​of the study such as the phase number, date, or sequence, and other information that can be interpreted from the drug information tables 900a-g.

[0072] Each of the drug information tables 900a to g is arranged within rows and columns. Table 900g includes graphical row markers 912 to separate adjacent rows, table 900c includes graphical column markers 914 to separate adjacent columns, and tables 900b, 900c, 900e, and 900f include both graphical row markers 912 and graphical column markers 914. As illustrated, graphical markers may include solid lines that are lighter (e.g., table 900c) or darker (e.g., table 900e) than the background color, abrupt changes in background color (e.g., table 900d), etc. Table 900a does not include graphical markers, and the other tables use graphical markers inconsistently, separating some rows or columns but not others. For example, table 900c includes graphical column markers 914 for each row except the top row. In some embodiments, each individual row in a data table may be associated with a single drug or candidate topic containing the information types described above. Any additional information associated with a drug or candidate topic may be presented in the table using a separate column in the form of name-value pairs. For example, a name-value pair may derive its name from header text associated with an identified column, and its value may be the context within an individual cell. In some embodiments, for example, if a column in a data table is identified as non-standard (i.e., not in the heading dictionary), the contents of that data table column may be stored in a database table column titled "Other" containing a list of name-value pairs. The format of that data may be [{'Name':'Column Name 1', Value':'Column Value 1'}, {'Name':'Column Name 2', Value':'Column Value 2'}]. In some embodiments, the database table may contain names that do not exist in a created dictionary of entity types. Additional information may also be stored in the form of metadata associated with data structuring, including data tables or technology distributions used to implement research or trials, among other data types.

[0073] In addition, each of the drug information tables 900a-g includes multiple progress bars 920 indicating the development stage of a given drug candidate (e.g., discovery, preclinical, Phase I, Phase II, Phase III, etc.). The progress bars 920 are illustrated in various styles and include arrows or bars of various shapes and colors. Generally, the progress bars 920 may span multiple cells.

[0074] In some embodiments, database tables may be created by defining new classification entities, which means defining columns or rows of data tables that have not been previously created as part of a manual heading dictionary. In this way, the system may create database-structured data entry templates based on the recognition of classification entities in newly identified image representations.

[0075] Figures 10A to 10D are simplified diagrams of drug information tables 1000a to 10D in which logical groups 1010 of text are automatically identified according to several embodiments. According to several embodiments consistent with Figures 1 to 8, the location of logical groups 1010 can be identified using a neural network model, as described above with reference to processes 302 to 304, for example. In particular, the annotations in tables 1000a to 1000 correspond to the output generated in process 304 of method 300. The depiction of drug information tables 1000a to 10D is determined using an experimental system having features consistent with system 100, and the experimental system is configured to carry out a method consistent with method 300. The neural network model used to identify the location of content objects corresponds to a YOLOv3 neural network model. Each of the logical groups 1010 is indicated with a border rectangle (dashed line) around the text.

[0076] Figures 11A to 11F are simplified diagrams of drug information tables 1100a to 11F according to several embodiments, in which cells are identified as belonging to specific rows and columns. The depictions of drug information tables 1100a to 11F were generated by the experimental system described above with reference to Figure 10. In particular, the annotations in tables 1100a to 1100 correspond to the output generated in process 305 of method 300. The location of a cell is identified by box 1110 (dashed line), and the location of a progress bar is identified by box 1120 (dashed line). According to several embodiments consistent with Figures 1 to 8, boxes 1110 and 1120 may be identified using one or more of process 305, method 500, and / or method 600. Arrows 1130 (dashed line) connect cells identified as belonging to a header row. Arrows 1140 (solid line) connect cells identified as belonging to a given column. Arrow 1150 (dotted line) connects cells and phase bars identified as belonging to a given row. According to several embodiments corresponding to Figures 1-8, arrows 1140-1150 may be identified using process 306, method 700, and / or method 800. In Table 1100b, the first column does not contain a header content object, so the default header cell 1160 is assigned to the first column.

[0077] The subject matter described herein may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, or in combination thereof, including the structural means and structural equivalents disclosed herein. The subject matter described herein may be implemented as one or more computer program products, such as one or more computer program products tangibly embodied in an information carrier (e.g., in a machine-readable storage device) or embodied in propagated signals, for execution by or control of a data processing device (e.g., a programmable processor, a computer, or multiple computers). A computer program (also known as a program, software, software application, or code) may be written in any form of programming language, including a compiled or interpreted language, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program may be stored in part of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinating files (for example, a file that stores one or more modules, subprograms, or parts of code). Computer programs may run on a single computer, on multiple computers in a single location, or be deployed distributed across multiple sites and interconnected by a communication network.

[0078] The processes and logic flows described herein, including the method steps of the subject matter described herein, may be implemented by one or more programmable processors that execute one or more computer programs to perform the functions of the subject matter described herein by operating on input data and producing outputs. The processes and logic flows may also be implemented by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the devices of the subject matter described herein may be implemented as special-purpose logic circuits, such as FPGAs or ASICs.

[0079] Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor will receive instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or will be operable to receive data from them, transmit data to them, or both. Information carriers suitable for embodying computer program instructions and data include, for example, all forms of non-volatile memory, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and optical disks (e.g., CDs and DVDs). Processors and memory may be complemented by or incorporated into special-purpose logic circuits.

[0080] To provide user interaction, the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device (e.g., mouse or trackball) on which the user can provide input to the computer. Other types of devices may similarly be used to provide user interaction. For example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, verbal, or tactile input.

[0081] The subject matter described herein may be implemented within a computing system including backend components (e.g., data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers having a graphical user interface or web browser for users to interact with implementations of the subject matter described herein), or any combination of such backend, middleware, and frontend components. The components of the system may be interconnected by digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0082] It should be understood that the disclosed subject matter is not limited in its application to the configuration details and the arrangement of components described or shown in the drawings below. Other embodiments of the disclosed subject matter are possible and can be carried out and implemented in various ways. It should also be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting.

[0083] Therefore, those skilled in the art will recognize that the underlying concepts of this disclosure can be readily used as the basis for designing other structures, methods, and systems to accomplish some of the objectives of the disclosed subject matter. It is therefore important that the claims be deemed to include such equivalent configurations, provided they do not deviate from the spirit and scope of the disclosed subject matter.

[0084] Although the disclosed subject matter has been described and illustrated in the exemplary embodiments described above, this disclosure is provided only as an example, and it should be understood that many changes in the details of the implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter, limited only by the following claims.

Claims

1. 1. A method comprising: accessing, by one or more computer processors, a pictorial representation of a data table, the data table including one or more cells arranged in one or more rows and one or more columns, the one or more cells including a first cell belonging to at least one first row and at least one first column, the first cell having a first content object inserted therein; providing, by the one or more computer processors, the image representation as an input to a neural network model that is trained to identify a location of a content object within the image representation; executing, by the one or more computer processors, the neural network model to identify a location of the first content object within the image representation; identifying, by the one or more computer processors, a location of the first cell based on the location of the first content object; determining, by the one or more computer processors, that the first cell belongs to the at least one first row and the first column based on one or more locations of the first cell and the first content object relative to a plurality of content objects associated with the one or more rows and the one or more columns; associating, by the one or more computer processors, the first content object with one or more classification identifiers based on a semantic analysis of the first content object; and populating, by the one or more computer processors, information associated with the first content object and information associated with the one or more classification identifiers into a structured database based on determining that the first cell belongs to the at least one first row and the at least one first column, the structured database including at least one data table row associated with the at least one first row and at least one data table column associated with the at least one first column; the location of the first content object includes a first region corresponding to at least a portion of the first content object; identifying the location of the first cell based on the location of the first content object; expanding the first region in at least one direction; determining that the expanded first region includes graphical markers marking one or more of row boundaries and column boundaries; in response to determining that the expanded first region includes the graphical marker, identifying the expanded first region as corresponding to the location of the first cell.

2. 2. The method of claim 1, further comprising: creating one of at least one second column and at least one second row in the structured database based on determining that the first cell does not belong to the at least one first row and the at least one first column.

3. Accessing the image representation comprises: receiving, by the one or more computer processors, a digital document over a computer network, the digital document including the data table; rendering, by the one or more computer processors, the digital document as a digital image; and locating, by the one or more computer processors, the pictorial representation of the data table within the rendered digital image.

4. Determining that the expanded first region includes the graphical marker includes: identifying a plurality of pixel locations corresponding to an edge of the expanded first region; determining, for each pixel location within the plurality of pixel locations, whether the pixel location is associated with a change in one or more of color and intensity along the at least one direction that exceeds a first predetermined threshold; determining that a count of the plurality of pixel locations associated with the change in color or intensity exceeds a second predetermined threshold; and determining that the expanded first region contains the graphical marker in response to determining that the count of the plurality of pixel locations exceeds the second predetermined threshold.

5. the location of the first cell includes a row span corresponding to a range of positions occupied by the cell along a row axis and a column span corresponding to a range of positions occupied by the cell along a column axis; determining, based on the location of the first cell, that the first cell belongs to the at least one first row and the at least one first column; sorting the one or more cells in the data table based on a plurality of locations of the one or more cells; and recursively performing operations, starting from a selected one of the one or more cells, for identifying one or more second cells belonging to the first row, the operations comprising: determining at least one other cell that has a row span that overlaps with the row span of the selected cell; identifying a cell among the at least one other cell that is closest to the selected cell; and identifying the closest cell as belonging to the at least one first row; selecting the closest cell as a next selected cell; identifying a header row of the one or more rows of the data table based on one or more header content objects populated into one or more header cells of the header row; determining that the column span of the first cell overlaps with a column span of a first header cell of the one or more header cells; identifying the first cell as belonging to the first column, the first column being associated with the first header cell; identifying the header row from among the one or more rows of the data table; generating one or more text representations corresponding to the one or more header content objects by optical character recognition; matching each of one or more textual expressions against a header dictionary, resulting in a score vector including one or more confidence scores corresponding to the one or more textual expressions, each confidence score being based on an extent of the match; determining a row score based on the score vector; selecting the header row based on the row score; The method of claim 1 , wherein determining a row score based on the score vector comprises calculating a sum of a square root of one or more of the score vector and the one or more confidence scores.

6. Selecting the header row comparing row scores associated with the one or more rows of the data table to one another; and selecting as the header row the row having the highest row score.

7. obtaining, by the one or more computer processors, a list of excluded content objects that are not eligible to be part of the header line; determining, by the one or more computer processors, whether the one or more rows of the data table include the excluded content object; 6. The method of claim 5, further comprising: identifying, by the one or more computer processors, the header row from the one or more rows of the data table from which rows containing the excluded content objects are excluded.

8. the first content object includes a graphical sequence object in which sequence information representing a progress or phase is illustrated using a progress bar; populating the structured database includes extracting the sequence information from the graphical sequence objects; The method of claim 1 , wherein the information associated with the first content object includes the sequence information.

9. 1. A computing system for populating a structured data set, comprising: accessing a pictorial representation of a data table, the data table including one or more cells arranged in one or more rows and one or more columns, the one or more cells including a first cell belonging to at least one first row and at least one first column, the first cell having a first content object inserted therein; providing the image representation as an input to a neural network model that is trained to identify a location of a content object within the image representation; executing the neural network model to identify a location of the first content object within the image representation; identifying a location of the first cell based on the location of the first content object; determining that the first cell belongs to the at least one first row and the first column based on one or more locations of the first cell and the first content object relative to a plurality of content objects associated with the one or more rows and the one or more columns; associating the first content object with one or more classification identifiers based on a semantic analysis of the first content object; and populating a structured database with information associated with the first content object and information associated with the one or more classification identifiers based on determining that the first cell belongs to the at least one first row and the at least one first column, the structured database including at least one data table row associated with the at least one first row and at least one data table column associated with the at least one first column; the location of the first content object includes a first region corresponding to at least a portion of the first content object; identifying the location of the first cell based on the location of the first content object; expanding the first region in at least one direction; determining that the expanded first region includes a graphical marker marking one of a row boundary and a column boundary; and in response to determining that the expanded first region includes the graphical marker, identifying the expanded first region as corresponding to the location of the first cell.

10. The processor, 10. The computing system of claim 9, further configured to create one of at least one second column and at least one second row in the structured database based on determining that the first cell does not belong to the at least one first row and the at least one first column.

11. Accessing the image representation comprises: receiving, by the one or more computer processors, a digital document over a computer network, the digital document including the data table; rendering, by the one or more computer processors, the digital document as a digital image; and locating, by the one or more computer processors, the pictorial representation of the data table within the rendered digital image.

12. Determining that the expanded first region includes the graphical marker includes: identifying a plurality of pixel locations corresponding to an edge of the expanded first region; determining, for each pixel location within the plurality of pixel locations, whether the pixel location is associated with a change in one or more of color and intensity along the at least one direction that exceeds a first predetermined threshold; determining that a count of the plurality of pixel locations associated with the change in color or intensity exceeds a second predetermined threshold; and determining that the expanded first region contains the graphical marker in response to determining that the count of the plurality of pixel locations exceeds the second predetermined threshold.

13. the location of the first cell includes a row span corresponding to a range of positions occupied by the cell along a row axis and a column span corresponding to a range of positions occupied by the cell along a column axis; determining, based on the location of the first cell, that the first cell belongs to the at least one first row and the at least one first column; sorting the one or more cells in the data table based on a plurality of locations of the one or more cells; starting from a selected one of the one or more cells, recursively performing an operation to identify one or more second cells belonging to the first row, the one or more second cells including the first cell, the operation comprising: determining at least one other cell that has a row span that overlaps with the row span of the selected cell; identifying a cell of the one or more cells that is closest to the first cell; identifying the closest cell as belonging to the at least one first row; selecting the closest cell as a next selected cell; identifying a header row of the one or more rows of the data table based on one or more header content objects populated into one or more header cells of the header row; determining that the column span of the first cell overlaps with a column span of a first header cell of the one or more header cells; identifying the first cell as belonging to the first column, the first column being associated with the first header cell; identifying the header row from among the one or more rows of the data table; generating one or more text representations corresponding to the one or more header content objects by optical character recognition; matching each of one or more textual expressions against a header dictionary, resulting in a score vector including one or more confidence scores corresponding to the one or more textual expressions, each confidence score being based on an extent of the match; determining a row score based on the score vector; selecting the header row based on the row score; 10. The computing system of claim 9, wherein determining a row score based on the score vector comprises calculating a sum of a square root of one or more of the score vector and the one or more confidence scores.

14. Selecting the header row comparing row scores associated with the one or more rows of the data table to one another; and selecting as the header row a row having a highest row score.

15. obtaining, by the one or more computer processors, a list of excluded content objects that are not eligible to be part of the header line; determining, by the one or more computer processors, whether the one or more rows of the data table include the excluded content object; 14. The computing system of claim 13, further comprising: identifying, by the one or more computer processors, the header row from the one or more rows of the data table from which rows containing the excluded content objects are excluded.

16. the first content object includes a graphical sequence object in which sequence information representing a progress or phase is illustrated using a progress bar; populating the structured database includes extracting sequence information from the graphical sequence objects; The computing system of claim 9 , wherein the information associated with the first content object includes the sequence information.

Images