Method and device for analysing structured data

Abstract

A computer-implemented method to identify relationships in a structured dataset. The method comprises for each column of a plurality of columns, determining column profile data associated with the column, wherein the column profile data comprises a set of distinct values of the plurality of values in the column. In response to comparing the column profile data of a first column with the column profile data of a second column, the first column associated with a first table of the plurality of tables and the second column associated with a second table of the plurality of tables, determining that the second column is a candidate match with regard to the first column. In response to determining the candidate match, creating a graph database edge between a table node associated with the first table and a table node associated with the second table.

Description

Method and device for analysing structured dataCross Reference to Related Application

[0001] The present application claims priority from Australian Provisional Patent Application No. 2023903653 filed on 14 November 2023, the contents of which are incorporated herein by reference in their entirety.Technical Field

[0002] Aspects of the disclosure relate generally to systems and methods for identifying relationships between data values of a stmctured dataset and, more specifically, to the representation of that relationship as an edge between nodes of a graph database.Background

[0003] Data management comprises the systematic organization, storage, retrieval, and protection of data assets within an organisation. Data management processes can encompass the entire data lifecycle, from data collection and entry to storage, maintenance, and eventual disposal. The primary goal of data management is to ensure data is accurate, accessible, secure, and relevant to support business operations and decision-making.

[0004] Data migration and data integration are desirable processes in the realm of data management. Data migration can include the process of transferring data from one system or format to another, typically with the aim of upgrading systems, or changing data storage methods. It is desired that data is migrated accurately, securely, and efficiently, while maintaining data integrity and availability.

[0005] Data integration focuses on unifying data from diverse sources, such as databases, applications, and cloud services, to provide a cohesive view of information. A desired outcome of data integration is to enhance data quality, enabling organizations to extract meaningful insights and support business processes.

[0006] The data integration process can identify relationships between the data that are not explicitly defined within the original data structures, e.g. previously undefined data relationships. The identification of previously undefined data relationships can provide an opportunity to leverage data as a strategic asset to empower data users to make informed decisions.

[0007] Data migration and integration processes can demand significant computational and memory resources, to collate and restructure the input data. Additionally, significant time investment is required by personnel experienced in the structure of the input data being migrated and integrated.

[0008] Accordingly, there is a need to provide a data integration method that ameliorates one or more of these difficulties, or other difficulties, of the prior art, or at least provides a useful alternative.

[0009] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is solely for the purpose of providing a context for the present invention. It is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention as it existed before the priority date of each claim of this application.Summary

[0010] In accordance with an aspect of the present disclosure, there is provided a computer- implemented method to identify relationships in a structured dataset. The structured dataset comprises a plurality of second-level data structures and a plurality of first-level data structures. Each first-level data structure comprises a plurality of values. Each first-level data structure is associated with a respective second-level data structure of the plurality of second-level data structures. The method comprises creating a table node for each second-level data structure of the plurality of second-level data structures. The method further comprises, for each first-level data structure of the plurality of first-level data structures, determining first-level data structure profile data associated with the first-level data structure, wherein the first-level data structure profile data comprises a set of distinct values of the plurality of values in the first-level data structure. The method further comprises, in response to comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first-level data structure profile data of a second first-level data structure of the plurality of first-level data structures, the first first-level data structure associated with a first second-level data structure of the plurality of second-level data structures and the second first-level data structure associated with a second second-level data structure of the plurality of second-level data structures, determining that the second first-level data structure is a candidate match with regard to the first first-level data structure. The method further comprises, in response todetermining the candidate match, creating a graph database edge between a table node associated with the first second-level data structure and a table node associated with the second second-level data structure.

[0011] In some embodiments, the method further comprises, for each first-level data structure of the plurality of first-level data structures: creating a column node; and for each distinct value of the plurality of distinct values in the first-level data structure: creating a value node; and associating the value node with the column node by a column-value edge.

[0012] In some embodiments, the column-value edge comprises a count of instances of the distinct value in the first-level data structure.

[0013] In some embodiments, the graph database edge indicates that the first second-level data structure is related to the second second-level data structure.

[0014] In some embodiments, the graph database edge represents a relationship between the first second-level data structure and the second second-level data structure that is undefined in the structured dataset.

[0015] In some embodiments, the method further comprises, for each first-level data structure of the plurality of first-level data structures: for each value of the plurality of values in the first- level data structure, in response to determining that the value is not in the set of distinct values, add the value to the set of distinct values.

[0016] In some embodiments, the method further comprises, determining first-level data structure profile data for each of a plurality of first-level data structures of the structured dataset to produce a plurality of sets of distinct values.

[0017] In some embodiments, the method further comprises, deduplicating the plurality of sets of distinct values to produce a superset of distinct values

[0018] In some embodiments, the method further comprises, creating a value node for each distinct value of the superset of distinct values.

[0019] In some embodiments, the first-level data structure profile data further comprises a count of instances of each distinct value of the plurality of distinct values in the first-level data structure.

[0020] In some embodiments, comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first-level data structure profile data of a second first-level data structure of the plurality of first-level datastructures comprises, determining that the first first-level data structure shares one or more distinct values with the second first-level data structure.

[0021] In some embodiments, comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first-level data structure profile data of a second first-level data structure of the plurality of first-level data structures further comprises one or more of: matching a data type of a distinct value of the first first-level data structure with a data type of a distinct value of the second first-level data structure; determining a number of distinct values shared by the first first-level data structure and the second first-level data structure; and determining a percentage of distinct values shared by the first first-level data structure and the second first-level data structure.

[0022] In some embodiments, determining that the second first-level data structure is a candidate match with regard to the first first-level data structure comprises determining one or more of: matching a data type of the first first-level data structure with a data type of the second first-level data structure; determining that a number of distinct values shared by the first first- level data structure and the second first-level data structure exceeds a threshold percentage; and determining that a number of distinct values shared by the first first-level data structure and the second first-level data structure exceeds a threshold amount.

[0023] In some embodiments, the method further comprises, in response to determining that the second first-level data structure is a candidate match with regard to the first first-level data structure, creating a candidate match edge between the first first-level data structure and the second first-level data structure.

[0024] In some embodiments, the candidate match edge comprises one or more of: a number of distinct values shared by the first first-level data structure and the second first-level data structure; a number of values in the first first-level data structure; a number of values in the second first-level data structure; a data type of the values in the first first-level data structure; and a data type of the values in the second first-level data structure.

[0025] In some embodiments, the method further comprises, for each first-level data structure of the plurality of first-level data structures: for each value of the plurality of values in the first- level data structure, creating a value node for the value; and determining structural information for the distinct value, wherein the structural information comprises an indication of the first- level data structure. In some embodiments, the method further comprises, creating a columnnode, based on the structural information associated with the distinct value; and creating an edge between the column node and the value node.

[0026] In some embodiments, the structured dataset comprises a relational database.

[0027] In accordance with an aspect of the present disclosure, there is provided a computer- implemented method to identify relationships in a structured dataset. The method comprises for each column of a plurality of columns, determining column profile data associated with the column, wherein the column profile data comprises a set of distinct values of the plurality of values in the column. In response to comparing the column profile data of a first column with the column profile data of a second column, the first column associated with a first table of the plurality of tables and the second column associated with a second table of the plurality of tables, determining that the second column is a candidate match with regard to the first column. In response to determining the candidate match, creating a graph database edge between a table node associated with the first table and a table node associated with the second table.

[0028] In accordance with an aspect of the present disclosure, there is provided a storage medium storing machine-readable instructions which, when executed by one or more processors, cause the one or more processors to perform a method described herein.

[0029] In accordance with an aspect of the present disclosure, there is provided a system comprising: one or more processors; and memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to perform a method described herein.Brief Description of Drawings

[0030] The embodiments of the disclosure will now be described with reference to the accompanying drawings, in which:Figure 1 illustrates a system for identifying relationships within a structured dataset, in accordance with an embodiment;Figure 2 is a schema of at least part of a relational database, in accordance with an embodiment;Figure 3 is a flowchart illustrating a process, performed by the controller, to identify relationships in a source dataset, in accordance with an embodiment;Figure 4 is a flowchart illustrating the relationship discovery algorithm, performed by the controller, in accordance with an embodiment;Figure 5 is a flowchart illustrating a method, performed by the controller, for creating graph database nodes, in accordance with an embodiment;Figure 6A illustrates an example table of data values of a source dataset, in accordance with an embodiment;Figures 6B illustrates a first example of a set of distinct values extracted by the controller from the table illustrated in Figure 6A, in accordance with an embodiment;Figures 6C illustrates a second example of a set of distinct values extracted by the controller from the table illustrated in Figure 6A, in accordance with an embodiment;Figure 7 illustrates an example graph database model, in accordance with an embodiment;Figure 8 illustrates a portion of a graph database created by the controller in response to performing method on the relational database of Figure 2 as a source dataset, in accordance with an embodiment;Figure 9 illustrates the graph database of Figure 8, as revised by the controller in response to discovering relationships between the table nodes, in accordance with an embodiment;Figure 10 illustrates a table of counts of distinct values, in accordance with an embodiment; andFigure 11 illustrates a graph database, determined by the controller from the “distinct values count” table of Figure 10, in accordance with an embodiment.Description of Embodiments

[0031] Data relationship discovery can include processes directed to identifying and understanding the connections, dependencies, and associations that exist among various data elements within a structured dataset.

[0032] Advantageously, data relationship discovery can assist users to derive valuable insights and use cases. For instance, it can be used to identify customer purchase patterns, optimize supply chain logistics, or uncover hidden insights in large datasets. Data relationship discovery may be particularly valuable in fields like data science, business intelligence, anddata analytics, where understanding the interconnectedness of data can lead to better decisionmaking, improved processes, and the development of predictive models.Graph databases

[0033] A graph database is a type of database configured to store relationships and connections between data points. A graph database typically represents data as nodes (entities) and edges (relationships) in a graph structure, enabling efficient modelling and traversal of complex, interconnected data. Graph databases are particularly suitable for arranging data that has many-to-many relationships, such as social networks, recommendation engines, and organisational hierarchies.

[0034] It may be desirable to combine multiple database systems to consolidate the data as a graph database in order to identify relationships within the data. Alternatively, or additionally, it may be desirable to reformat a structured dataset from a non-graph database structure to a graph database structure such that data relationships within the dataset may be identified. In particular, it may be desirable to convert one or more relational databases into a graph database.

[0035] Systems and methods provided herein receive a source dataset, comprising structured data, as an input, and apply data migration and integration techniques to output the data from the source dataset as a graph database. The data migration and integration techniques include a relationship discovery algorithm configured to discover undefined relationships between data structures in the source dataset, and indicated these relationships within the outputted graph database.Source dataset structures

[0036] As described herein, a source dataset comprises a structured dataset, in which data is arranged in accordance with a data structure. The data structure may comprise a hierarchy of data structure levels. Preferrably, the data structure defines at least a first-level data structure and a second-level data structure, such that data values are arranged into a plurality of first- level data structures and the first-level data structures are arranged into a plurality of second- level data structures.

[0037] A structured dataset may be configued in accordance with a defined data structure formats, such as: a relational database; a graph database; comma separated variables (CSV); JavaScript Object Notion (JSON) structure; Extensible Markup Language (XML) format and Apache Parquet format. A source dataset may comprise a pluralty of structured datasets, whichmay be configured in accordance with one or more different data structure formats. For example, a source dataset may comprise a relational database and a set of CSV files.

[0038] A relational database is a structured data storage system that uses a tabular format to organize data. A common format for a relational database comprises tables with rows and columns, where each row represents a specific record, and each column defines a data attribute. In a situation in which the source dataset comprises a relational database, the first-level data structures comprise columns, and the second-level data structures comprise tables.

[0039] A JSON structure may comprise a file, which in turn comprises one or more objects, which in turn comprise one or more names, and each name is associated with a value. In the situation in which the source dataset comprises a JSON structure, the first-level data structures comprise names, and the second-level data structures comprise files.

[0040] For clarity of reference herein, first-level data structures are referred throughout as columns, and second-level data structures are referred throughout as tables.Determining data relationships

[0041] Data relationships may be explicitly defined with the schema of a source dataset. Alternatively, or additionally, data relationships may be explicitly defined within documentation associated with the database. However, documentation may be incomplete, inaccurate or out of date. Often, the identification of data relationships within the data of a source dataset relies on existing human knowledge, which can have gaps.

[0042] The data in a source dataset may also comprise undefined data relationships. An undefined data relationship is not explicitly defined within the database scheme or within the database documentation. The discovery of these undefined data relationships may be particularly beneficial for building knowledge and meaning of the data within the source dataset. Advantageously, serendipitous discovery of meaning within the data may occur, based on the discovery of previously undefined data relationships.Using JOINs for relationship discovery

[0043] Relational databases enable efficient data retrieval, searching, and reporting through SQL (Structured Query Language) queries. Data records of a relational database are linked logically by a foreign key. To discover undefined data relationships in a relational data storage model, complex queries, such as SQL join operations, may be built to perform actions over and over again. Join operations provide a computational means to combine data from multiple tables based on shared attributes or keys.

[0044] The join operation produces joined data from the selected tables, including columns from multiple tables, with rows that match based on the specified key. The joined data may be analysed to determine relationships and connections between the data points.

[0045] Join operations can be resource intensive in terms of computational time, computational resources (e g., processor time, memory requirements) and personnel time, especially when there are several tables joined, millions of rows on tables, or complex join queries that traverse various levels through subqueries. Additionally, join-intensive query performance in relational databases tends to deteriorate as the dataset gets bigger.

[0046] In some situations, the task of identifying relationships in the data may be dropped because it is computationally too costly, or it appears to be intractable. Similarly, in some situations, the task of data relationship discovery is performed to a limited extent due to limited resources. Lack of knowledge of data relationships within a body of data can cause delays and functional limitations to data migration or integration tasks.Example - Computational cost of join operations

[0047] In some embodiments, the number of column join operations that may be performed to scan all tables within a structured dataset, with a view to discovering undefined data relationships, is a binomial coefficient function:Join operationsthe number of datatypes across the structured dataset; n is the average number of columns per table; c is the number of different coefficients i is the summation index, starting at 1 r is the length of the coefficient - the number of columns in each join

[0048] In some embodiments, r is equal to 2, but in other embodiments, r can be more. In most embodiments, r is always 2 to start with. The number of join operations performed for multi-way join candidates (including 3, 4, or 5 columns to be matched) may be determined by adjusting variable c.

[0049] In one example, assuming there are 5 datatypes (string, float, integer, date, datetime), with a single coefficient (2), if the average number of compatible columns per datatype is 500,then the estimated number of join operations to be performed is around 0.62 million. In another example, if the average number of compatible columns per datatype is 1000, then the estimated number of join operations to be performed is around 2.5 million.

[0050] In another example, in which there are 3 coefficients (2, 3 and 4 columns to validate), if the average number of compatible columns per data type is 500, then the estimated number of join operations to be performed is around 13 billion. Similarly, if the average number of compatible columns per data type is 1000, then the estimated number of join operations to be performed is around 41 billion.

[0051] Accordingly, in many situations, the number of join operations to be performed to identify potential relationships within the data of a relational database is so large as to be practically infeasible to perform and prohibitively computationally expensive.

[0052] Systems and methods provided herein may ameliorate the computational expense associated with identifying undefined data relationships in a structured dataset by applying a relationship discovery algorithm that does not utilise join operations.Figure 1 - System architecture

[0053] Figure 1 illustrates a system for identifying relationships within a structured dataset, in accordance with an embodiment. The system 100 may comprise one or more device(s) 101 which are in communication with external data storage 122 and a server 124 over a network 120.

[0054] The device 101 may comprise a mobile or handheld computing device such as a smartphone or tablet, a laptop, or a PC, and may, in some embodiments, comprise multiple computing devices.

[0055] The device 101 comprises a controller 110. The controller 110 may be comprised of one or more processors. A processor may comprise one or more microprocessors, microcontrollers, central processing units (CPUs), application specific instruction set processors (ASIPs), application specific integrated circuits (ASICs), or other component capable of reading and executing instruction code.

[0056] The one or more processors of the controller 110 are, in combination or individually, configured to execute program code 180 stored within the system memory 112. The code defines an application to be executed by the controller. When executed by the controller, the code causes the device to function according to the described embodiments.

[0057] The system memory 112 stores device specific data, which may include configuration information which defines the function of the device 101.

[0058] The device 101 may comprise a user interface (UI) 145. In embodiments, the user interface 145 comprises a means for the user to input data to the application 180. The means for the user to input data may comprise one or more of a peripheral device such as a keyboard or a mouse, a touch screen, or other means of the user providing input to the application 180.

[0059] The device may comprise one or more displays 140. Each of the one or more displays 140 may be configured to display a graphical user interface in implementing a method, such as that illustrated in Figure 3. In embodiments, the user interface 145 utilises the graphical user interface displayed on the display 140 as a means for the application 180 to output data to a user.

[0060] Application 180 may be executed, in part or in full, on device 101. Application 180 may be executed, in part or in full, on one or more servers 124 in communication with the device. Machine-readable code (e g. software) defining application 180 may be stored, in part or in full, on device 101. Machine-readable code (e g. software) defining application 180 may be stored, in part or in full, on server(s) 124. The application 180 may store the output products in data storage 122, memory 112, and / or transmit the output products over network 122.

[0061] The communications interface 118 may comprise a combination of network interface hardware and network interface software suitable for establishing, maintaining and facilitating communication over a relevant communication channel.

[0062] The network 120 may include, for example, at least a portion of one or more networks having one or more nodes that transmit, receive, forward, generate, buffer, store, route, switch, process, or a combination thereof, etc. one or more messages, packets, signals, some combination thereof, or so forth. The network 120 may include, for example, one or more of: a wireless network, a wired network, an internet, an intranet, a public network, a packet-switched network, a circuit-switched network, an ad hoc network, an infrastructure network, a public- switched telephone network (PSTN), a cable network, a cellular network, a satellite network, a fibre-optic network, some combination thereof, or so forth.Figure 2 - Example relational database schema

[0063] Figure 2 is a schema 200 of at least part of a relational database, in accordance with an embodiment. The relational database may comprise at least part of the source dataset(s) 302.

[0064] The relational database schema 200 defines six tables, which are each configured to store data pertaining to a company and the employees thereof. The schema defines the data types of the data stored in each table. For example, the schema defines a table 202 for storing information pertaining to the departments of the company. The departments table 202 comprises two columns, namely a department name column (represented in the schema by dept_name 220), and a department number column (represented in the schema by dept_no 222). In other words, the department name column 220 is associated with the departments table 202, and the department number column 222 is associated with the departments table.

[0065] The tables of the schema 200 are linked via keys, which define how a table relates to another table. For example, the department manager table 206 is related to departments table 202 via the department number 214. The department number value is stored in both the department manager table and the department table.Figure 3 - Method

[0066] Figure 3 is a flowchart illustrating a process 300 performed by the controller 110 to identify relationships in a source dataset, in accordance with an embodiment. Process 300 receives as input a source dataset 302, and outputs at least a portion of a graph database 340.

[0067] The source dataset comprises structured data. The source dataset may comprise: one or more relational databases; one or more JSON files; one or more Parquet files; one or more other data structures; or a combination thereof.

[0068] The outputted graph database 340 may comprise a graph database, or a portion thereof.

[0069] In broad terms, process 300 comprises three operations, namely: an input scoping operation 310; a relationship discovery algorithm 320; and a knowledge creation and consolidation operation 330.

[0070] In operation 310, the controller 110 performs preliminary work to scope the source dataset 302 to identify the components of the source dataset. In operation 320, the system applies the relationship discovery algorithm to the one or more source datasets. The relationship discovery algorithm is described further in relation to Figure 4.

[0071] In operation 330, a user utilises the relationships discovered by applying the relationship discovery algorithm 320.Input scoping operation

[0072] The preliminary work of the input scoping operation 310 may comprise obtaining access to the one or more source datasets; determining the size and structure of the data within the source dataset; determining the schema, structure or format of the data in the source dataset.

[0073] In embodiments, the scoping operation 310 may comprise one or more of: obtaining a list of applications; obtaining a list of schemas in each application; obtaining a list of tables in each schema; obtaining a list of views in each schema; and obtaining a list of data files (e g. JSON, Parquet).

[0074] In some embodiments, the scoping operation 310 may further comprise validation steps, in which the system validates whether the tables described in the dataset schemas exist and are accessible. The scoping operation 310 further comprises determining one or more columns within each table of the plurality of tables within the one or more datasets.Figure 4 - Method

[0075] Figure 4 is a flowchart illustrating the relationship discovery algorithm, in accordance with an embodiment. The relationship discovery algorithm is described in relation to the example source dataset 302, which is defined by example schema 200. Additionally, the relationship discovery algorithm is described in relation to an example instance of table 230, as illustrated in Figure 6A. Figures 6B and 6C illustrates respective examples of a set of distinct values 250 as extracted by the controller in an instance of operation 322 performed on table 230, in accordance with an embodiment.

[0076] For clarity of explanation, the example schema 200, and the example table 230, constitute a simple use case comprising only a small quantity of data. It is to be noted, however, that the methods described herein may be applied to real enterprise data, which comprises a substantially larger quantity of data, and for which it is infeasible to computationally discover undefined relationships, and / or infeasible to discover undefined relationships through human or machine consideration of the source dataset.

[0077] The controller 110 performs data extraction operations 321, 322 and 323 for each table identified in operation 310. Alternatively, in some embodiments, the controller may perform data extraction operations 321, 322 and 323 for a subset of the tables identified in operation 310.

[0078] In operation 321, the controller determines the plurality of columns within the table. With reference to table 230, the controller determines that the plurality of columns comprises:an employee number column ‘emp_no’ 602; a date column ‘to_date’ 604; and a travel expenses amount column ‘expense_amt’ 606.

[0079] For each column in the plurality of columns determined in operation 321, the controller performs operation 322. In operation 322, the controller considers the set of values within the column and determines the set of distinct values 350 from the set of values within the column. The controller stores the set of distinct values in memory 130 for subsequent use by the controller. In embodiments, the set of distinct values may be stored as a list, a table, or another format interpretable by the controller.Distinct values

[0080] A first value is considered to be distinct from a second value if the first value differs from the second value. A first value may differ from a second value in terms of its numerical value, alphanumerical content, format, or other attribute. If a first value is the same as a second value, then the first value and the second value are considered to be duplicates, rather than distinct values. In response to the controller determining that a first value is a duplicate of a second value, the controller stores only one of the first or second values within the set of distinct values.

[0081] The determination of whether a value is the same as another value may depend on the application to which operation 322 is applied. The controller 110 may be configured to ignore leading or trailing white space when determining whether a string value is distinct from another string value. Similar, the controller may be configured to ignore case (upper or lower case) when determining whether a string value is distinct from another string value.

[0082] With reference to column 602 of table 230, the controller determines the set of distinct values as 610. With reference to column 604 of table 230, the controller determines the set of distinct values as 612.

[0083] In some embodiments, the controller stores a set of distinct values for each column, of each table considered by the relationship discovery algorithm 400.

[0084] The set of distinct values may be stored in a suitable format, e.g., a comma separated file, a table, a JSON file, a graph database, or any combination thereof.Structural information

[0085] The controller stores the set of distinct values 350 in association with data structure information that indicates the column from which the controller determined the distinct value.For example, the data structure information may indicate: the column from which the controller determined the distinct value; the table to which that column belongs; the schema to which that table belongs; and the application to which that schema belongs.

[0086] Preferably, for each distinct value of the set of distinct values 350, the controller determines structural information associated with that distinct value.

[0087] The structural information comprises information identifying the particular data structure hierarchy associated with the distinct value. For each distinct value, the structural information identifies the data structure entities to which a distinct value belongs. The structural information may be stored in association with each distinct value in the set of distinct values 350.

[0088] For example, in the embodiment illustrated in Figure 4, for each distinct value in the set of distinct values 350, the structural information comprises column information indicating the column from which the distinct value was extracted. Similarly, the structural information may comprise information indicating: the table to which that column belongs; the schema to which that table belongs; and the application to which that schema belongs.Column data metrics

[0089] In operation 323, the controller 110 determines column data metrics 360 for the column. The column data metrics 360 may comprise an indication of the documented datatype of the values in the column. A datatype may comprise datatypes such as ‘text’ or ‘integer’

[0090] In some situations, the values in the column may appear to be a different datatype than the documented datatype for that column. Accordingly, the controller 110 may determine an actual datatype of the values in a column, and the column data metrics 360 may comprise an indication of the actual datatype of the values in that column.

[0091] In some embodiments, the column data metrics comprise an indication of the numerical range of the values in the column. In some embodiments, the column data metrics comprise a classification of the type of data in the column, as suggested by the values in the column. For example, the classification may indicate a phone number, an address, a last name, a product code.

[0092] In some embodiments, the column data metrics comprise an indication of max, min, median, mode, standard deviation, and / or average for numerical fields. In some embodiments, the column data metrics comprise an indication of a commonly occurring value pattern (e.g., sequence of letters, numbers, punctuation, white space). In some embodiments, the column datametrics comprise an indication of the least commonly occurring value pattern, and / or the most commonly occurring value pattern.

[0093] In some embodiments, the column data metrics comprise an indication of uniqueness of the values within the column (e.g., a percentage of values found in the column that are unique). In some embodiments, the column data metrics comprise an indication of the total number of values in the column.

[0094] In operation 324, the controller may utilise some or all of the column data metrics to determine candidate matches between columns.Figure 5 - Deduplicate values

[0095] In operation 325, the controller 110 is configured to create graph database nodes based on the distinct values determined in operation 322. Figure 5 is a flowchart illustrating a method, performed by the controller, for creating graph database nodes, in accordance with an embodiment.

[0096] The controller is configured to create database nodes from a superset of distinct values 355. The superset of distinct values comprises the sets of distinct values 350 as determined by operation 322 for each column of the source dataset 302.

[0097] Each value in each set of distinct values 350 is locally distinct, with respect to the column from which the set of distinct values was derived. However, tables and columns within a dataset may share many values. Accordingly, the superset of distinct values may contain duplicate values when considering all tables together.

[0098] Preferably, these duplicated values are deduplicated before the value nodes are created to avoid having the same value represented more than once in the graph database. If duplicates are loaded, the controller may fail to identify column to column relationships.

[0099] In some embodiments, the controller is configured to de-duplicate values in the superset of distinct values 355, to remove instances of non-distinct values within the superset of distinct values. De-duplicating the superset of distinct values produces a revised superset of distinct values 355. In response to the controller de-duplicating the values in the superset of distinct values, the values in the revised superset of distinct values are all distinct within that revised superset of distinct values 355.Consolidating structural information

[0100] In the de-duplication process, distinct values that are duplicates across a plurality of sets of distinct values 350, are de-duplicated, such that only one instance of the distinct value remains in the superset of distinct values.

[0101] In one embodiment, the structural information associated with each of the duplicate distinct values are consolidated, such that there are two sets of structural information stored in association with the deduplicated distinct value in the superset of distinct values. Accordingly, it can be determined that a distinct value was located in two different columns within the source dataset(s) 302.Creating nodes

[0102] In operation 325, the controller is configured to create a graph database node for each distinct entity within the source datasets 302.

[0103] In the example illustrated in Figure 5, the source datasets comprise a data structure which includes: values; columns; tables; schemas; and applications. Each distinct value, table, column, schema and application is considered to be a distinct entity, for which the controller is configured to create a graph database node.

[0104] The controller 110 is configured to determine the distinct entities of the source dataset(s) based on the structural information stored in association with each distinct value of the superset of distinct values 355

[0105] With reference to Figure 5, the controller 110 determines a list of applications from the structural information associated with each distinct value in the superset of distinct values 355. The controller de-duplicates this list of applications to determine a list of distinct applications 356.

[0106] Similarly, the controller determines a list of schemas from the structural information associated with each distinct value in the superset of distinct values 355, and de-duplicates the list of schemas to determine a list of distinct schemas 354. The controller determines a list of tables from the structural information associated with each distinct value in the superset of distinct values 355, and deduplicates the list of tables to determine a list of distinct tables 353. The controller determines a list of columns from the structural information associated with each distinct value in the superset of distinct values 355, and de-duplicates the list of columns to determine a list of distinct columns 352.

[0107] In accordance with the example illustrated in Figure 5, the controller creates a graph database node for: each distinct value; each distinct column; each distinct table; each distinct schema; and each distinct application. The controller loads the newly created nodes into a graph database 340.

[0108] In embodiments, the controller is configured to apply slowly changing dimension techniques when loading the created nodes into the graph database. Slowly changing dimension techniques may allow users of the graph database 340 to determine a history of changes to the database nodes over time.Relationships / Edges

[0109] Graph database nodes are connected via graph database edges, which are also referred to herein as edges or relationships. Graph database edges comprise one or more edge properties which characterise and quantise the relationship between the nodes joined by the edge.

[0110] In operation 325, the controller 110 is configured to determine one or more edges between the nodes created in operation 325. The controller is configured to determine the edges based on the structure of the source dataset 302. For example, as described below, the controller is configured to determine one or more edges for the nodes of graph database 340, based on the relational database schema 200.

[0111] The controller is configured to load the edges into the graph database. The determination and loading of edges may occur in conjunction with the creation of the nodes that form the graph database. In some embodiments, the controller is configured to load column to value edges using slowly changing dimension techniques.Figure 7 - Example graph database model

[0112] The structure of a graph database may be defined by a graph database model which describes types of nodes within the graph database and types of edges that connect the nodes.

[0113] Figure 7 illustrates an example graph database model 700, in accordance with an embodiment. In operation 375, the controller is configured to create a graph database in accordance with the graph database model 700.

[0114] The graph database model 700 defines the node types of the graph database, namely value nodes 702, column nodes 704, table nodes 706, schema nodes 708 and application nodes 710. The graph database model further describes the relationships (e.g., edges) that may exist between specific nodes of the node types.Belongs to relationship

[0115] In operation 325, the controller is configured to determine, create and load ‘belongs to’ relationships into the graph database 340.

[0116] In accordance with the embodiment illustrated in Figure 7, a column ‘belongs to’ 714 a table; a table ‘belongs to’ 716 a schema; and a schema ‘belongs to’ 718 an application. One or more column nodes may belong to a table node. Similarly, one or more table nodes may belong to a schema node, and one or more schema nodes may belong to an application node.Is found in relationship

[0117] In operation 325, the controller is configured to determine, create and load ‘is found in’ relationships into the graph database 340.

[0118] In response to the structural information associated with a distinct value indicating that the distinct value was extracted from a distinct column, the controller creates an ‘is found in’ relationship 712 between the node of the distinct value and the node of that distinct column.

[0119] A distinct value may be located more than one column of the source dataset(s) 302. These columns may not all belong to the same table. Rather, the columns may each belong to different tables. Accordingly, a value node be connected to a plurality of column nodes, wherein each connection comprises a different ‘is found in’ relationship 712.Figure 8 - Example graph database

[0120] Figure 8 illustrates a portion of a graph database 800 created by the controller in response to performing method 300 with relational database 200 as a source dataset, in accordance with an embodiment. The illustrated portion of the graph database comprises only the table nodes. For clarity, nodes representing values, columns, schemas and applications are not illustrated in Figure 8.

[0121] The portion of the graph database 800 comprises six nodes, one for each table of the relational database represented by schema 200. Node 802 was created by the controller 110 in operation 325 and represents the departments table 202 of the relational database. The graph database may further comprise column nodes (not shown in Figure 8), such as column nodes corresponding to the department name 220 and the department number 222. The graph database may further comprise value nodes for each distinct value in the department name column and the department number column of the relational database.

[0122] The keys of the relational database schema 200 describe how one table of the relational database relates to another table of the relational database (e.g., the relationship between one table and another table). Each key of the relational database describes a known data relationships between the tables of the relational database.

[0123] In operation 325, the controller creates edges between the tables nodes to represent the known data relationships between tables. Edge 814 represents a known relationship between the department manager table node 806 and the departments table node 802. The edge 812 comprises edge properties which define the nature of the relationship between the department manager table node 806 and the departments table node 802. In particular, the edge properties indicate that each data entry of the department manager table ‘is a manager of a data entry of the departments table. The edge properties of edge 816 indicate that each data entry of the employees table 810 ‘is managed by’ a data entry of the department manager table 806

[0124] Notably, the node 830 representing the travel_expenses table is not connected, by an edge, to any other node shown in Figure 8. Accordingly, the relationship between the travel_expenses table and the other tables represented in the graph database is unknown at this stage.Shares value with relationship

[0125] In operation 324, the controller is configured to determine whether a column shares a value with another column. In some embodiments, a column is considered to share a value with another column when at least one of the distinct values (in the superset of distinct values 355) was located within both columns.

[0126] In response to determining that a column shares a value with another distinct column, the controller creates a ‘shares value’ relationship 722 between the node of the first distinct column and the node of the second distinct column.

[0127] In some embodiments, the controller determines that that a column node shares a value with another column node by considering the structural information associated with the superset of distinct values 355. In some embodiments, the controller determines that a column node shares a value with another column node by considering the other ‘is found in’ relationships 712 associated with the value nodes that are connected to the first column node via an ‘is found in’ relationship.

[0128] A column node may be connected to a plurality of other column nodes via a plurality of respective ‘shares value’ relationships.

[0129] In some embodiments, the edge properties of a ‘shares value with’ relationship between a first column node and a second column node indicate the number of values shared between a first column and the second column. In other words, the edge properties of a ‘shares value with’ relationship between a first column node and a second column node indicate the number of times both the first column and the second column have an ‘is found in’ relationship with a value node. The number of times both the first column and the second column have an ‘is found in’ relationship with a value node may be correlated with the strength of the relationship between the first column and the second column.Determining counts of distinct values

[0130] In some embodiments, in operation 324, the controller is configured to determine a count of distinct values in one or more columns of the source dataset. Figure 10 illustrates a table 1010 of counts of distinct values, which have been determined by a controller 110 of system 100 performing operation 324, in accordance with an embodiment.

[0131] In this embodiment, the controller applies a SQL count function against columns of tables in a source dataset. The controller writes the determined SQL-count output to the distinct values count 1010. The distinct values count table 1010 has columns for “column name”, “value”, “count of that value in the column”. The SQL-count output for each column is added to the distinct values count table, producing a single list of values found in the columns of the source dataset. The compound primary key for the distinct values count 1010 includes column_name and value, and others to enable the data to be cleanly updated.

[0132] The controller may use the distinct values count table 1010 to create the is_found_in relationships in the graph database. In step 322, the controller 110 identifies the relationships between values and columns of table 1010 to create a graph database.

[0133] Figure 11 illustrates a graph database 1110, determined by the controller, from the distinct values count table 1010, in accordance with an embodiment. Graph database 1110 illustrates a plurality of is_found_in relationships created from the distinct values count table 1010. In particular, graph database 1110 comprises a value node 1102, which represents the value ‘100’; a value node 1104, which represents the value ‘99’; a column node 1106, which represents the column ‘cust_score’; a column node 1108, which represents the column ‘credit_score’; and a column node 1112, which represents the column ‘default_credit_score’.

[0134] In some embodiments, all other relationships that the controller identifies from the source dataset are built on top of the is_found_in relationships determined from the distinctvalues count 1010. For example, from the is_found_in relationships illustrated in the graph database 1110, one can deduce the following:1292 occurrences of value “100” are found in the cust_score column;27652 occurrences of value “100” are found in the credit_score column; and958 occurrences of value “100” are found in the default_credit_score column.

[0135] Furthermore, one can deduce: (100 -> cust_score) * 1292 (100 — > credit_score) * 27652 (100 — > default_credit_score) * 958.

[0136] Accordingly, the controller can determine a shares_values_with relationships from the cust_score column node 1106 to each of the credit_score column node 1108 and the default_credit score column node 1112, and a shares_values_with relationship from the credit_score column node 1108 to the default_credit_score column node 1112.

[0137] The number of the shares_values_with relationships created between column nodes of the graph database 1110 indicates the strength of the relationship between the columns associated with the column nodes. The controller may use the strength of the relationship between the columns to determine candidate matches between columns (e.g. in operation 324).

[0138] Advantageously, by determining the distinct values count table 1010, and determining the is_found_in relationships and the shares_values_with relationships, a system may review a significantly smaller amount of data in comparison to the large volume of data of the source dataset (e.g. source dataset 302).

[0139] Advantageously, count operations are typically a low cost operationally in terms of computational resources and memory access resources. Additionally, count operations may leverage existing metadata in the database catalogue Even when full table scans are required to count values in columns, the operation may consume numerous disk in / out operations, but the compute operations remain trivial. Advantageously, the determination of distinct values count tables, via count operations, avoids the use of join operations, which are typically applied in conventional methods. Join operations are notably expensive in terms of consuming computational resources and system resources. Accordingly, the avoidance, or minimisation, of the use of join operations may provide substantial savings in computational and system resources and may therefore contribute towards providing a highly responsive user experience.

[0140] The controller may be configured to retain one or more distinct value count tables in memory (such as system memory 112) such that the one or more distinct value count tablesmay be recalled at a later date. Additionally, the controller may be configured to retain an indication of the scope of the source dataset that has been counted to produce the one or more distinct value count tables.

[0141] In the event that the source dataset is modified to incorporate additional content, the controller may be configured to re-perform operation 324, in which the controller retrieves the stored one or more distinct value count tables and updates the tables to incorporate counts of the values of the additional content of the source dataset. Accordingly, the determination and storage of distinct value count tables provides an opportunity to reused calculated resources, and to ameliorate the computational costs incurred when the source dataset is updated or modified.

[0142] In contrast to defining relationships between columns via relational database structures, in which values must be stored for each column, duplicating the underlying data many times without benefitting from the natural deduplication in a graph structure, the determination of is_found_in and shares_values_with relationships require reduced storage resources.Determining a candidate match

[0143] In operation 324, the controller determines candidate matches between distinct columns extracted from the source dataset 302. In particular, the controller determines whether a column node is a candidate match with another column node. The determination of a ‘candidate match’ relationship between a first column node and a second column node suggests that there is an undiscovered relationship between the first column node and the second column node. An undiscovered relationship may comprise a relationship between columns of the source dataset that is not defined by the structure of the source dataset and / or not defined by the schema of the source dataset.

[0144] The controller may consider a number of factors when determining whether a column node is a candidate match with another column node. In some embodiments, the controller considers the presence of a ‘shares value with’ relationship existing between a first column node and a second column node to be a prerequisite for determining that the first column node is a candidate match with the second column node.

[0145] The set of distinct values 350 and the column data metrics 360, in combination, comprise column profile data 370. A controller may be configured to consider the column profile data 370 associated with a first column and the column profile data associated with asecond column to determine whether the first column is a candidate match with the second column.

[0146] In particular, the controller may be configured to consider any one or a combination of the following factors when considering whether a first column is a candidate match with a second column: a documented datatype of values in the first or second column; an actual data type of values in the first or second column; a percentage of values found in the first column that are shared by both the first column and the second column; a percentage of values found in the second column that are shared by both the first column and the second column; a count of the values shared by both the first column and the second column; whether a number of distinct values shared by the first column and the second column exceeds a threshold percentage; whether a number of distinct values shared by the first column and the second column exceeds a threshold amount; and whether the values of the first column are a subset of the values of the second column, or vice versa.

[0147] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the shared distinct value count between a first column and the second column. The shared distinct value count comprises an indication of the number of distinct values shared by the first column and the second column. The shared distinct value count may indicate the strength of the relationship between the first column and the second column.

[0148] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the percentage of distinct values from the first column that are also found in the second column. In some embodiments, the controller is configured to consider the percentage of distinct values from the second column that are also found in the first column. A larger percentage of shared values may indicate a stronger relationship between the first column and the second column.

[0149] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the number of distinct values in the first column. In some embodiments, to determine a candidate match, the controller is configured to consider the number of distinct values in the second column. A higher number of distinct values may be more predictive of a relationship between the columns, than a lower number of distinct values.

[0150] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider a total number of values (including duplicate values) from the first column that match values from the second column. In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider a total number of values (including duplicate values) from the second column that match values from the first column. This total number of values may be larger than the number of matching distinct values, as it includes duplicate values.

[0151] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider a number of distinct values that are present in the first column, but are not present in the second column. In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider a number of distinct values that are present in the second column, but are not present in the first column. This number can indicate the cardinality between the first column and the second column.

[0152] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the datatype that is most frequently inferred in the first column and the datatype that is most frequently inferred in the second column. If the inferred datatypes of the first and second columns are not the same, then the controller may be configured to consider the first column is not a candidate match with the second column.

[0153] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the number of times that the most frequently inferred datatype for the first column is present in the first column. Similarly, for the second column, the controller may be configured to consider the number of times that the most frequently inferred datatype for the second column is present in the second column. For example, in there are 100 values in a first column but the inferred datatype count is 33, it suggests that most values in the column are not of the modal datatype. Not sharing modal datatypes, in this case, may not be a useful indicator of whether a candidate match is present between the first column and the second column.

[0154] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider the average length of values in the first column and / or the second column. Average lengths of approximately 4-32 tend to indicate keys. Accordingly, this consideration may be used to filter for potential keys.

[0155] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider a data pattern associated with values present in the first column and / or values present in the second column. The data pattern may comprise the most frequently identified pattern in values found in the column. A pattern implies some semantic meaning of the data, such as a phone number, identification code or date. For example, ‘+99 (9)99 999 999’ suggests an Australian formatted phone number, or ’99 Aaa 9999’ suggests a date’. The controller may be configured to consider a candidate match is present based on a number of times that a data pattern present in the first column is also present in the second column, and / or vice versa.

[0156] In some embodiments, to determine a candidate match between a first column and a second column, the controller is configured to consider input provided by one or more users indicating the presence or an absence of a relationship between the first column and the second column.Candidate match relationship

[0157] In response to determining that a first column is a candidate match with a second column, the controller creates and loads a ‘candidate match’ relationship 720 between the first column node and the second column node in the graph database 340.

[0158] Advantageously, the controller may determine candidate matches based on the data that has already been extracted by the controller from the source dataset 302 during the migration of the source dataset to a graph database 340 in operations 321, 322 and 323. Accordingly, in embodiments, the controller does not issue further queries to the source dataset in order to determine the candidate matches.

[0159] Additionally, advantageously, the computational complexity of determining candidate matches based on the extracted data (e.g. superset of distinct values 355 and the column data metrics 360) may be substantially less than the computational complexity of performing multiple join operations on the source dataset to determine the same or similar candidate matches.Relates to relationship

[0160] In operation 326, the controller determines previously unknown relationships (e g., discovered relationships) between table nodes, based on the candidate matches determined in operation 324. The discovered relationships are referred to as a ‘relates to’ relationship, andcomprise a newly discovered relationship that was not defined in the structure of the source dataset 302.

[0161] The controller creates edges between the table nodes of the graph database 340 to represent a ‘relates to’ relationship. A first table may relate to a second table when the first table has at least one column that is a candidate match with a column in the second table. Accordingly, a table node 706 may have a ‘relates to’ relationship 724 with another table node of the graph database.

[0162] In operation 326, the controller 110 may also create one or more ‘relates to’ relationships between nodes of the same entity type, to identify previously undiscovered relationships at the higher levels of the graph database model 700. Accordingly, the controller may create ‘relates to’ relationships 726 between schema nodes, and ‘relates to’ relationships 728 between application nodes.Indicating discovered relationships

[0163] In some embodiments, the controller is configured to indicate to a user of the system 100, the discovered relationship between table nodes (e.g., a discovered relationship between second-level data structures of a source dataset), via the user interface 145. Indicating the discovered relationship may comprise displaying, via the user interface the graph database, or part thereof, and visually highlighting the discovered relationship. Indicating the discovered relationship may comprise outputting a description of the discovered relationship Indicating the discovered relationship may comprise indicating a strength of the discovered relationship.Figure 9 - Adding discovered relationships

[0164] Figure 9 illustrates the graph database of Figure 8, as revised by the controller in response to performing operations 324 and 326 to discover relationships between the table nodes, in accordance with an embodiment.

[0165] Notably, the relationship discovery algorithm is configured to identify relationships between nodes of the graph database that are not formally document in the source dataset and may not be readily identifiable by considering the source dataset, or by applying a computation method such as JOINs to identify undiscovered relationships.

[0166] In operation 324, the controller determines that the employee number (emp_no) column 602 of the travel_expenses table 230 ‘shares values with’ the employee number column of the department manager 206 table. Furthermore, in response to determining that the valueswithin the employee number column of the travel_expenses table are a subset of the values within the employee number column of the department manager table, the controller determines that the employee number column of the travel_expenses table is a candidate match to the employee number column of the department manager table.

[0167] In response to determining this candidate match, the controller creates a graph database edge 814 between the travel_expenses node 830 and the department manager node 806.Edge thickness

[0168] In some embodiments, in an illustration of a graph database, the thickness of a line representing an edge between two nodes can provide an indication of an attribute of the relationship between those two nodes. In some embodiments, the thickness of the lines connecting two table nodes may be proportional to the number of column relationships that the node pairs share.

[0169] The thickness of the edge 850 connecting dept_manager to titles, indicates that there are 5 columns in each table that are candidates to match with each other. The edge 860 connecting titles to employees is thinner and indicates there is only one column in each table is a candidate for matching.Security measures

[0170] In some embodiments, the controller is configured to convert the data values of the source dataset to a cryptographic hash. The cryptographic hash of the data values may be stored in the value nodes of the graph database 340.

[0171] By storing a one-way cryptographic hash of the data value to users the hash may be used as a database key instead of the value. Advantageously, the data values may be kept hidden from users. Additionally, a one-way hash may be applied such that any time the controller sees the same data value within the database, the controller can create a matching hash to update the database without using the actual value.

[0172] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. Furthermore, it will be appreciated by persons skilled in the art that embodiments disclosed herein can be combined with one or more other embodiment disclosed herein, without departing from the broad general scope of the presentdisclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

[0173] It will be appreciated by persons skilled in the art that any suitable distribution of functionality between different functional units may be used without detracting from the invention. For example, functionality illustrated to be performed by separate computing devices may be performed by the same computing device. Likewise, functionality illustrated to be performed by a single computing device may be distributed amongst several computing devices. Hence, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.

[0174] It will be appreciated by persons skilled in the art that, for processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in differing order. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations can be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

[0175] References herein to software or executable instructions are to be understood as referring to executable instructions stored in volatile or non-volatile memory. The memory can include any data storage device that can store data which can thereafter be read by a processor. Examples of memory include read-only memory (ROM), random-access memory (RAM), magnetic tape, optical data storage device, flash storage devices, or any other suitable storage devices.

[0176] Throughout this specification the word ‘comprise’, or variations such as ‘comprises’ or ‘comprising’, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0177] As used herein, any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. Similarly, use of “a” or “an” preceding an element or component is done merelyfor convenience. This description should be understood to mean that one or more of the element or the component is present unless it is obvious that it is meant otherwise.

[0178] Unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0179] Unless expressly stated to the contrary, the description of an entity as a first entity or a second entity is used to distinguish one entity from another entity and does not constitute an implied or explicit ordering of the referenced entities.

Claims

CLAIMS:

1. A computer-implemented method to identify relationships in a structured dataset, the structured dataset comprising a plurality of second-level data structures and a plurality of first-level data structures, each first-level data structure comprising a plurality of values, each first-level data structure associated with a respective second- level data structure of the plurality of second-level data structures; the method comprising: creating a table node for each second-level data structure of the plurality of second- level data structures; for each first-level data structure of the plurality of first-level data structures: determining first-level data structure profile data associated with the first-level data structure, wherein the first-level data structure profile data comprises a set of distinct values of the plurality of values in the first-level data structure; and in response to comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first-level data structure profile data of a second first-level data structure of the plurality of first- level data structures, the first first-level data structure associated with a first second- level data structure of the plurality of second-level data structures and the second first-level data structure associated with a second second-level data structure of the plurality of second-level data structures: determining that the second first-level data structure is a candidate match with regard to the first first-level data structure; and in response to determining the candidate match, creating a graph database edge between a table node associated with the first second-level data structure and a table node associated with the second second-level data structure.

2. The method of claim 1, further comprising, for each first-level data structure of the plurality of first-level data structures: creating a column node; andfor each distinct value of the plurality of distinct values in the first-level data structure: creating a value node, and associating the value node with the column node by a column-value edge.

3. The method of claim 2, wherein the column-value edge comprises a count of instances of the distinct value in the first-level data structure.

4. The method of any of claims 1 to 3, wherein the graph database edge indicates that the first second-level data structure is related to the second second-level data structure.

5. The method of any of claims 1 to 4, further comprising, in response to determining that the second first-level data structure is the candidate match with regard to the first first- level data structure, determine that there is a relationship between the first first-level data structure and the second first-level data structure.

6. The method of claim 5, further comprising outputting, to a user, an indication of the relationship.

7. The method of any of claims 5 to 6, wherein the graph database edge represents the relationship between the first second-level data structure and the second second-level data structure.

8. The method of any of claims 5 to 7, wherein the relationship that is undefined in the structured dataset.

9. The method of any of claims 1 to 8, further comprising, for each first-level data structure of the plurality of first-level data structures: for each value of the plurality of values in the first-level data structure, in response to determining that the value is not in the set of distinct values, add the value to the set of distinct values.

10. The method of any of claims 1 to 9, further comprising, determining first-level data structure profile data for each of a plurality of first-level data structures of the structured dataset to produce a plurality of sets of distinct values.

11. The method of claim 10, further comprising, deduplicating the plurality of sets of distinct values to produce a superset of distinct values.

12. The method of claim 11, further comprising, creating a value node for each distinct value of the superset of distinct values.

13. The method of any of claims 1 to 12, wherein the first-level data structure profile data further comprises a count of instances of each distinct value of the plurality of distinct values in the first-level data structure.

14. The method of any of claims 1 to 13, wherein comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first-level data structure profile data of a second first-level data structure of the plurality of first-level data structures comprises: determining that the first first-level data structure shares one or more distinct values with the second first-level data structure.

15. The method of claim 14, wherein comparing the first-level data structure profile data of a first first-level data structure of the plurality of first-level data structures with the first- level data structure profile data of a second first-level data structure of the plurality of first- level data structures further comprises one or more of: matching a data type of a distinct value of the first first-level data structure with a data type of a distinct value of the second first-level data structure; determining a number of distinct values shared by the first first-level data structure and the second first-level data structure; and determining a percentage of distinct values shared by the first first-level data structure and the second first-level data structure.

16. The method of any of claims 1 to 15, wherein determining that the second first-level data structure is a candidate match with regard to the first first-level data structure comprises determining one or more of: matching a data type of the first first-level data structure with a data type of the second first-level data structure;determining that a number of distinct values shared by the first first-level data structure and the second first-level data structure exceeds a threshold percentage; and determining that a number of distinct values shared by the first first-level data structure and the second first-level data structure exceeds a threshold amount.

17. The method of any of claims 1 to 16, further comprising, in response to determining that the second first-level data structure is a candidate match with regard to the first first-level data structure, creating a candidate match edge between the first first-level data structure and the second first-level data structure.

18. The method of claim 17, wherein the candidate match edge comprises one or more of: a number of distinct values shared by the first first-level data structure and the second first-level data structure; a number of values in the first first-level data structure; a number of values in the second first-level data structure; a data type of the values in the first first-level data structure; and a data type of the values in the second first-level data structure.

19. The method of any of claims 1 to 18, further comprising, for each first-level data structure of the plurality of first-level data structures: for each value of the plurality of values in the first-level data structure, creating a value node for the value; and determining structural information for the distinct value, wherein the structural information comprises an indication of the first-level data structure.

20. The method of claim 19, further comprising: creating a column node, based on the structural information associated with the distinct value; and creating an edge between the column node and the value node.

21. The method of any of claims 1 to 20, wherein the structured dataset comprises a relational database.

22. A storage medium storing machine-readable instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1 to 21.

23. A system comprising: one or more processors; and memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to perform the method of any of the preceding claims.

Images