System and method for ontology induction by statistical profiling and reference schema matching

Abstract

To provide a data artificial intelligence system and a data AI system used in data flow management, and data integration for utilizing machine learning dataflow machine learning used for building a complex dataflow application and a pipeline or other computing environments.SOLUTION: A process for generating a functional system on the basis of one or more generating rules comprises steps of: accepting input defining a reference HUB; acquiring one or more entity definitions associated with a dataset or an entity provided by the reference HUB; generating sample data thereof; determining one or more metrics associated with the sample data by profiling the sample data; generating one or more rules on the basis of the entity definition; generating a functional system on the basis of the rules; and persisting the functional system and the sample data profile for use in processing the data input.SELECTED DRAWING: Figure 47

Claims

1. 1. A computer-implemented method comprising: The method includes a step of: a processor of the computer accessing metadata stored in a memory, the metadata describing a data source and a data target, the data source including a data set or an entity; and the method further includes: said processor accessing a schema describing entity definitions within said data source or data target; generating one or more rules based on entity definitions associated with the data set or entities; wherein generating one or more rules includes using data rules derived by performing ontology analysis to determine relationship rules that define correspondences between the datasets or entities in the data source or data target and constraints or references imported from a reference schema that includes an ontology defined based on relationships between the datasets or entities and their attributes, and the method further comprises: The method includes the processor generating a functional system based on the generated one or more rules.

2. The step of determining the relationship rules comprises: said processor accessing further schemas describing entity definitions in further data sources or data targets; and using the data rules derived by performing the ontology analysis, the processor further determines relationship rules that define correspondences between data sets or entities in other data sources or data targets.

3. The method described in claim 1, wherein the metadata specifies a data transformation for receiving data from the data source or providing data to the data target, and the data transformation is a mapping based on relationships with other datasets or entities.

4. 10. The method of claim 1, wherein the method is performed to at least one of receive the data from one or more data sources offered as cloud services in a cloud computing environment, or output the data to one or more data targets offered as cloud services in the cloud computing environment.

5. a processor; and a storage device for storing metadata, the metadata describing data sources and data targets, the data sources including datasets or entities; The processor: accessing a schema describing entity definitions within said data source or data target; generating one or more rules based on entity definitions associated with the data set or entities; generating the one or more rules includes using data rules derived by performing ontology analysis to determine relationship rules that define correspondences between the datasets or entities in the data source or data target and constraints or references imported from a reference schema that includes an ontology defined based on relationships between the datasets or entities and their attributes; The processor generates a functional system based on the generated one or more rules.

6. determining the relationship rules accessing further schemas describing entity definitions in further data sources or data targets; and the processor uses the data rules derived by performing the ontology analysis to determine relationship rules that define correspondences between data sets or entities in further data sources or data targets.

7. 6. The system of claim 5, wherein the metadata stored in the knowledge source and describing one or more data sources and data targets specifies data transformations for receiving data from the data sources or providing data to the data targets, the data transformations being mappings based on relationships with other data sets or entities.

8. 6. The system of claim 5, wherein the system is used to at least one of receive the data from one or more data sources offered as cloud services in a cloud computing environment, or output the data to one or more data targets offered as cloud services in the cloud computing environment.

9. A program for causing a computer to execute the method according to any one of claims 1 to 4.

Images