Near real-time data cataloging and deep lineage for massive data discovery

Abstract

A data platform that provides large-scale data discovery with near-real-time data cataloging and detailed lineage. Data is received from one or more sources and written to a data storage system. The received data is processed in the data storage system using sensors in the data storage system. Metadata and data definitions associated with the data are sent to a data service in near-real time. The data service generates a data catalog based on the metadata and data definitions associated with the data. Data lineage for the data is created in the data service. The data catalog and data lineage are presented in a data discovery interface.

Description

[Technical Field]

[0001] This specification relates to near real-time data cataloging and large-scale data discovery with detailed lineage, and methods for using same. [Background technology]

[0002] Organizations have many teams that generate data. To be able to leverage this data, users want to search across the entire set of data available within the enterprise. Data lineage tools show the evolution of data over time through metadata. Data lineage identifies the flow of data. Data lineage can identify the creation of the data, the changes made to the data, and the final target that contains the data. Thus, data lineage identifies where the data came from, what processes were involved in creating the data, who was involved in its creation and transformation, and where the data currently resides.

[0003] A data catalog uses the same information to create a searchable inventory of data assets within an organization. Using a data catalog and lineage, users can discover the data that exists in an organization. Data objects have attributes such as who the owner is, what is the structure of the data object, who produced the data object, who is consuming the data object downstream, and what downstream process is used to produce the data object.

[0004] There are numerous components or products, such as Apache Atlas, that provide different aspects of data discovery, data cataloging, and lineage. Apache Atlas provides native support for Hadoop-based systems. Apache Ranger® is a framework for enabling, monitoring, and managing comprehensive data security across the Hadoop platform. Another product, Atlan, combines metadata from various sources to create a unified data discovery, cataloging, lineage, and governance experience across data assets. Additional components, such as Spark, use connectors with Apache Atlas to help build data lineage.

[0005] However, these individual components are standalone and do not provide complete user configuration, for example, from generating data to generating a data catalog to querying and returning datasets. Additionally, existing components do not provide native integration with all tools within the data platform to help users make informed business decisions, do not enable visualization of complex data transformations within lineage, or provide lineage and catalog optimization to reduce data and process duplication and reduce resource utilization. Summary of the Invention [Problem to be solved by the invention]

[0006] In at least an embodiment, a method for providing a data platform includes receiving data from one or more sources, writing the received data to a data storage system, using sensors at the data storage system to process the data received at the data storage system, transmitting metadata and data definitions associated with the data to a data service in near real time, generating a data catalog at the data service based on the metadata and data definitions associated with the data, creating data lineage for the data at the data service, and presenting the data catalog and data lineage in a data discovery interface.

[0007] In at least one embodiment, the data platform includes a memory that stores computer-readable instructions and a processor coupled to the memory, the processor configured to execute the computer-readable instructions to perform operations, the operations including: receiving data from one or more sources; writing the received data to a data storage system; using sensors in the data storage system to process the data received at the data storage system; transmitting metadata and data definitions associated with the data to a data service in near real time; generating, at the data service, a data catalog based on the metadata and data definitions associated with the data; creating, at the data service, data lineage for the data; and presenting the data catalog and data lineage in a data discovery interface.

[0008] In at least one embodiment, a non-transitory computer-readable medium stores computer-readable instructions that, when executed by a processor, cause the processor to perform operations including: receiving data from one or more sources; writing the received data to a data storage system; using sensors at the data storage system to process the data received at the data storage system; transmitting metadata and data definitions associated with the data to a data service in near real time; generating, at the data service, a data catalog based on the metadata and data definitions associated with the data; creating, at the data service, data lineage for the data; and presenting the data catalog and data lineage in a data discovery interface.

[0009] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying figures. It should be noted that, according to standard industry practice, various features have not been drawn to scale. In fact, the dimensions of various features may be increased or decreased for clarity of illustration. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a data platform according to at least one embodiment.

[0011] [Figure 2] FIG. 2 illustrates a process of a data platform according to at least one embodiment.

[0012] [Figure 3A] FIG. 3A shows a data computer interface without a synchronous mode.

[0013] [Figure 3B]FIG. 3B illustrates a data computing interface with a synchronous mode, according to at least one embodiment.

[0014] [Figure 4] FIG. 4 illustrates high availability for Apache Atlas application pods, according to at least one embodiment.

[0015] [Figure 5] Figure 5 shows the simple data lineage provided by existing components.

[0016] [Figure 6] FIG. 6 illustrates capturing discrete processes in detailed data lineage, according to at least one embodiment.

[0017] [Figure 7] FIG. 7 illustrates data pattern detection and optimization in accordance with at least one embodiment.

[0018] [Figure 8] FIG. 8 is a flowchart of a method for providing near real-time data cataloging and large-scale data discovery with detailed lineage, according to at least one embodiment.

[0019] [Figure 9] FIG. 9 is a high-level functional block diagram of a processor-based system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] The embodiments described herein illustrate examples of implementing different features of the provided subject matter. To simplify the disclosure, example components, values, operations, materials, arrangements, and the like are described below. Of course, these are examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, and the like are contemplated. For example, the formation of a first feature above or on a second feature in the following description includes embodiments in which the first and second features are formed in direct contact with each other, and also includes embodiments in which an additional feature is formed between the first and second features such that the first and second features are not in direct contact with each other. Additionally, the present disclosure repeats reference numbers and / or letters in various examples. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations described.

[0021] Additionally, spatially relative terms such as "beneath," "below," "lower," "above," and "upper" are used herein for ease of description to describe the relationship of one element or feature to another element(s) or feature(s), as shown in the figures. Spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation shown in the figures. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0022] Terms such as "user equipment," "mobile station," "mobile," "mobile device," "subscriber station," "subscriber equipment," "access terminal," "terminal," "handset," and similar terms refer to wireless devices utilized by subscribers or users of wireless communication services to receive or transmit data, control, voice, video, sound, games, data streams, or signaling streams. The foregoing terms are used interchangeably in this specification and related drawings. Terms such as "access point," "base station," "Node B," "evolved Node B (eNode B)," next generation Node B (gNB), enhanced gNB (en-gNB), Home Node B (HNB), "Home Access Point (HAP)," and the like refer to components or devices of a wireless network that source and receive data, control, voice, video, sound, games, data streams, or signaling streams from UEs.

[0023] In at least an embodiment, a method for providing a data platform includes receiving data from one or more sources, writing the received data to a data storage system, using sensors at the data storage system to process the data received at the data storage system, transmitting metadata and data definitions associated with the data to a data service in near real time, generating a data catalog at the data service based on the metadata and data definitions associated with the data, creating data lineage for the data at the data service, and presenting the data catalog and data lineage in a data discovery interface.

[0024] According to at least one embodiment, the data platform receives input from one or more sources at an authentication and middleware interface. The data platform includes a data governance, data storage layer, a data computation interface, a data discovery interface, and a query interface. A data computation interface, a data service interface, and a query interface are provided. The data computation interface includes a data creation and transformation layer that writes data to or reads data from the data storage layer. The data governance (access control layer) controls access to the data storage layer by determining access rights to the data storage layer. New data and changes are captured by sensors in the data storage layer and pushed to the data computation interface. The data computation interface pushes metadata, creates data lineage, and performs data audits. This information, along with data definitions, is pushed to the data service interface. The data service interface includes a data discovery interface that presents the data catalog and data lineage. Duplicates in the data lineage and data catalog are determined and eliminated. Users can access the data discovery interface in the data service interface to search and discover data of interest across the data. The data computation interface pulls metadata change events. Data lineage is created to present detailed, granular lineage information, including details for use in performing complex data transformations. The data services interface receives input at the data discovery interface to query the metadata associated with the data in the data storage system. Tag data, classification data, and categorization data are associated with the data.In response to a query being generated by the data query interface, a dataset is identified by the data query interface in the data storage layer based on input to the data discovery interface of the data service interface and the metadata and data definitions fetched from the data service interface. A data governance (access control layer) controls access to the data storage layer by requesting access rights to the data storage layer. In response to the access rights being granted, a user can generate a query to retrieve the identified dataset. Data is then fetched directly from the data storage layer via the data query layer.

[0025] Accordingly, the embodiments described herein provide certain advantages, including one or more of: providing a one-stop solution for the data usage lifecycle; enabling end users to discover data in near real-time with minimal manual intervention; enabling informed business decisions through low-level lineage including detailed procedures and other metadata details, including schemas, related to data transformations; providing ease of use through automated creation of a data catalog and integration with the query layer; or provisioning appropriate data semantics and accountability. The pattern detection feature helps any user discover / search for any data based on the type of action (or set of actions) that led to the generation of that particular data. This knowledge helps users / applications reduce their costs and efforts by avoiding duplication of datasets and the actions used to generate them.

[0026] FIG. 1 illustrates a data platform 100 according to at least one embodiment.

[0027] 1 , authentication and middleware interface 110 receives input from one or more sources 112, for example, via web UI 114 and / or onboarding interface 116. Data computation interface 120 includes a data generation and transformation layer that writes or reads data to data storage layer 130. Pipeline configuration and initiation 117 and data migration flow initiation 118 are configured to ingest data at data computation interface 120. Data computation interface 120 processes and distributes data across multiple resources using Spark / NiFi jobs 122 and pulls that data from stream queues using data definition capture jobs 124. Thus, data computation interface 120 can read / write data 126 via data storage layer 130.

[0028] The data storage layer 130 implements a data storage system that stores the datasets. Different data storage technologies can be implemented, such as MinIO 131, Yugabyte 132, MySQL® 133, Kafka 134, etc. Data Definition Language (DDL) streams 136 present data to the data storages 131-134, and sensors 138 classify the data into new and changed data and capture change information that is pushed to the data computing interface 120 and then to the data service interface 140. The data governance (access control layer) 150 controls access to the data storage layer 130. The data computing interface 120 pulls metadata change events 128 from the data storage layer 130.

[0029] The data services interface 140 exposes a data discovery interface 142, a data catalog 144, and data lineage 146. An access control list (ACL) plugin 147 provides rules for access to the data storage layer 130 for reading / writing data 148. The data query interface 160 includes a query-as-a-service (QaaS) 162 that allows users to process and query data. The virtual warehouse layer 164 provides an abstracted view of the data in the data storage as well as tools and APIs to extract the data.

[0030] A data discovery interface 142 is accessed by a user wishing to search and discover 170 data of interest across the data in the data services interface 140. Data owners can tag data and create 171 classifications and categorizations associated with that data. A data query interface 160 allows for the creation of 172 views on existing data and supports queries 173 across multiple data stores. First-level analytical reports 174 can be created based on queries associated with datasets identified in the data storage.

[0031] The data computation interface 120 pushes metadata, creates data lineage, and performs data audits 180 to the data services interface 140. In at least one embodiment, metadata and data definitions associated with data received from one or more data storage systems are provided to the data services interface 140 in near real-time. Those skilled in the art will recognize that non-real-time refers to a response time of one second or greater, and near-real-time refers to a response down to 10 milliseconds (ms). The data query interface 160 fetches the metadata and data definitions 184 from the data services interface 140. Based on the accepted query, the data query interface 160 can receive read query data 166 for retrieval.

[0032] FIG. 2 illustrates the processes of a data platform 200 according to at least one embodiment.

[0033] 2, authentication and middleware interface 210 receives input from one or more sources 212, for example, via web UI 214 and / or onboarding interface 216. Data computation interface 220 includes a data generation and transformation layer that writes data to or reads data from data storage layer 230. Data governance (access control layer) 250 controls access to data storage layer 230 by determining access rights to data storage layer 230.

[0034] At "1", configuration and initiation of pipeline 217 and initiation of data migration flow 218 are configured to capture data into data sources in data storage layer 230 where sensors 238 capture these changes. The data is provided to data computation interface 220. Data computation interface 220 includes a data generation and transformation layer that writes data to or reads data from data storage layer 230.

[0035] The data computation interface includes Spark / NiFi jobs 222 to pull that data from the stream queue and data definition capture jobs 224. The data definition capture jobs 224 include pre-configured jobs for data definition capture that pull data from the stream queue. The data definition capture jobs 234 extract information and convert it into a unified format understood by the data discovery interface 224 of the data services interface 240.

[0036] At "2", data is read and written 226 to the data storage layer 230 by the data computation interface 220. The data storage layer 230 includes multiple data storage systems for storing datasets. Different data storage technologies can be implemented, such as MinIO 231, Yugabyte 232, MySQL 233, and Kafka 234. The data storage layer 230 stores datasets using Data Definition Language (DDL) streams 236. DDL is a subset of Structured Query Language (SQL), a language for describing data and its relationships in databases, such as MinIO 231 and Yugabyte 232. An ACL plugin 247 provides rules for access to the data storage layer 230 for reading / writing 248 data. A query interface 360 ​​includes a Query-as-a-Service (QaaS) 262 that allows users to query data. The virtual warehouse layer 264 provides an abstracted view of the data in the data storage as well as tools and APIs to extract the data.

[0037] At "3," the sensor 238 data is written to a database that categorizes the data into new and changed data and captures change information that is pushed to the data computing interface 220 and then to the data services interface 240. The sensor 238 detects changes in the data and then pushes the changes to the data computing interface 220 and then to the data services interface 240. For example, a file is written to a database, and the sensor 238 sends the file to the data computing interface 220. A metadata capture service pulls these and creates a data catalog 244 and a data lineage 246. The data lineage 246 is created to present detailed, granular lineage information, including details such as data classification and business topology tagged in metadata, for use in performing complex data transformations.

[0038] At "4", the data computing interface 220 pulls the metadata change event 228.

[0039] At "5", the data computing interface 220 pushes metadata, creates data lineage, and audits data 280 in near real time to the data services interface 240. Data definitions can also be provided to the data services interface 240 by the data computing interface 220.

[0040] The data services interface 240 includes a data discovery interface 242, which presents a data catalog 244 and data lineage 246. In response to a file being created from an existing file, data lineage 246 is captured that identifies the file as derived from a prior existing file. Metadata information 280 is presented to the data services interface 240 by the data computation interface 220 in a unified discovery format that is subsequently provided to the duplicate sensor.

[0041] In "6", duplicates in the data lineage and data catalog 281 are eliminated. New data paths that share the same job are added to the data lineage 246, and the data paths are associated under one data catalog 244 based on metadata and data definitions. The data catalog 244 presents a catalog related to information such as lineage, data path, owner, and data description in data discovery 242.

[0042] Data services interface 240 allows users to search and discover 270 data of interest across the data by accessing data discovery interface 242. Data lineage and data auditing 280 is pushed from data computing interface 220 to data services interface 240. Data services interface 240 receives input to query metadata associated with data in data storage systems. Tag data and created classifications and categorizations 271 are associated with the data.

[0043] The data query interface 260 allows for the creation of views 272 on existing data and supports queries across multiple data stores 273. The data service interface 240 contains the data definitions that are provided to the data query interface 260.

[0044] When a data catalog 244 or data object is pushed to the data services interface 240, the data is optimized before the data catalog 244 is created. For example, one or more users attempt to process the same data twice. An existing pipeline used to create a computational job to write the data can exist while a second pipeline associated with the same data is created. However, one data lineage 246 is created. The data discovery interface 242 determines that the data already exists in the storage systems 231-234, and the data storage layer 230 does not create anything new again.

[0045] At "7", metadata and data definitions are fetched from data services interface 240 for data catalog 244. Data query interface 260 fetches metadata definitions 284 from data services interface 240 and generates corresponding tables or views. Data query interface 260 implements applications used to search data storage layer 230 across data catalog 244. A user can also query any data catalog 244 through data query interface 260. When a query is generated by data query interface 260, data sets are identified by data query interface 260 in data storage layer 230 based on input to data discovery interface 242 of data services interface 240 and the metadata and data definitions fetched from data services interface 240. Data governance (access control layer) 250 controls access to data storage layer 230 by requesting access rights to data storage layer 230.

[0046] In response to the access permissions for the identified dataset, the query interface 260 generates a query to retrieve the identified dataset based on the input, the metadata, and the data definition. The data storage layer 230 increases the availability of data discovery for data in the data storage system in response to the query and includes a proxy layer that processes search requests for and responses from multiple applications under the control of a central coordinator in the data storage layer 230, which identifies an active leader from the applications and enables the proxy layer to forward requests to and receive responses from the active leader. Data is then fetched directly from the data sources in the data storage layer 230.

[0047] At "8," analytical first-stage reports 274 are generated based on queries associated with the identified datasets in the data storage. Pattern recognition and classification algorithms run in the background based on the data catalog 244 and data lineage 246. Data discovery 242 and data catalog 242 of data services interface 240 provide mechanisms to help a user or application identify existing dataset patterns by suggesting initial possible matches for datasets in data storage layer 230 and data lineage 246. Thus, searching data catalog 244 includes crawling data catalog 244 and data lineage 246 and presenting suggestions for possible matches of similar datasets or similar sets of actions to perform to generate similar datasets.

[0048] At "9", the queried data is read 266.

[0049] 3A and 3B illustrate the addition of a synchronous mode to a data computing interface, according to at least one embodiment.

[0050] FIG. 3A illustrates a data computer interface without a synchronous mode 300. In FIG. 3A, the operation of existing tools such as Apache Spark 310 is illustrated. Apache Spark 310 provides a data processing and analysis engine that implements stage-oriented scheduling using jobs and stages. New datasets are created in response to processing jobs on existing data. Data is captured in a consistent manner. However, existing tools such as Apache Spark 310 suffer from inconsistent data lineage due to asynchronicity, resulting in some data lineage not being captured.

[0051] As shown in Figure 3A, Spark Context 310 reflects jobs that run on data and generate new data points. To capture transformation steps being applied to the data, Spark 310 is implemented with connectors that push process details to a data discovery interface. Events 320 from DAG Scheduler 322, SQL Execution 324, Streaming Execution 326, etc. are pushed to Live Listener Bus 312. Listener 312 is the component that is notified when an event occurs. DAG Scheduler 322 translates the logical execution plan (RDD lineage of dependencies built using RDD transformations) into a physical execution plan (using stages). Resilient Distributed Datasets (RDDs) are Spark's fundamental data structure: immutable, distributed collections of objects. A Spark job is a computation sliced ​​into a set of parallel tasks or stages. SQL Execution 324 produces SQL execution metrics. Streaming Execution 326 produces batches of data.

[0052] The asynchronous event queue 314 receives events 320 from an event handler, such as an all event posting process 316. The event postings 330 are provided to a per-thread event list 332. Multiple listener threads 334 are provided, and events from the event list 332 are provided to the appropriate listener threads in the multiple listener threads 334.

[0053] However, due to the asynchronous nature of the asynchronous event queue 314, data lineage is not consistently captured. The live listener bus 312 allows tracking of events that Spark emits while the application is running. Events 320 are typically application start / end, job start / end, stage start / end, etc. (see links). Listeners are spawned as child threads outside the Spark Context 310. This means that when the Spark main thread completes its function and the function is still being executed by the listener thread 334, the thread may be killed because it is an asynchronous event that the Spark main thread does not wait for to complete.

[0054] FIG. 3B illustrates a data computer interface in a synchronous mode 350 according to at least one embodiment.

[0055] The Spark Context 360 reflects the jobs that are running on the data and generating new data points. To capture the transformation steps being applied to the data, Spark 360 is implemented with connectors that push process details to the data discovery interface. Events 370 from the DAG scheduler 372, SQL execution 374, streaming execution 376, etc. are pushed to the live listener bus 362.

[0056] An asynchronous event queue 364 receives events 320 from event handlers, such as the All Event Posting Process 366. A synchronous mode has been added where the Spark main thread waits for listener threads to complete. Events from the DAG scheduler, SQL execution, streaming execution, etc. are pushed to the live listener bus. In addition to the asynchronous event queue 364, a synchronous event queue 367 can also receive events 320 from event handlers, such as the All Event Posting Process 366. The synchronous event queue 367 ensures that data lineage is captured before the job completes.

[0057] Multiple instances of the Apache Atlas connector are active. An additional layer is added on top to check status and provide a response so that the active connector connected to the Atlas connector can be determined. Synchronization events are provided to a synchronization event list 368. From the synchronization event list 368, synchronization events are provided to multiple synchronization listeners 369. An active connector is requested and the connections are checked until an active connector is identified.

[0058] The event posting 380 is then provided to a per-thread event list 382. Multiple listener threads 384 are provided, and events from the event list 382 are provided to appropriate listener threads within the multiple listener threads 384.

[0059] FIG. 4 illustrates high availability for an Apache Atlas application pod 400, according to at least one embodiment.

[0060] In FIG. 4 , Apache Atlas 410 is used to provide aspects of data discovery, data cataloging, and data lineage. Apache Atlas 410 provides native support for Hadoop-based systems. Apache Atlas 410 can be deployed on Google Kubernetes (GK) Cluster, Amazon Kubernetes Cluster, plain Kubernetes Cluster, or any other Kubernetes-based cluster. In FIG. 4 , instances of Apache Atlas 410 are provided as shown in pod 1 412, pod 2 414, and pod 3 416. Apache Atlas 410 can have more or fewer instances than pod 1 412, pod 2 414, and pod 3 416. Apache Atlas is coupled to Yugabyte 420 and a search engine 430, such as Apache Solr. Apache Atlas 410 uses and interacts with various systems to provide metadata management and data lineage. Apache Solr 430 is used to index Atlas data so that the data can be searched.

[0061] In response to a failover from an active instance to a standby instance on Kubernetes, Apache Atlas becomes unreachable. To address this issue and make Apache Atlas cloud-compatible, a proxy layer 440 is used to interface between Apache Atlas 410 and users 450 and external jobs 452. A central coordinator, such as Zookeeper 460, is used to manage access to the active instance. Zookeeper 460 is a centralized service for providing configuration, naming, synchronization, and group services across large clusters in a distributed system.

[0062] Zookeeper460 provides the infrastructure for cross-node synchronization by maintaining in-memory state-type information on Zookeeper460. Zookeeper460 keeps a copy of the system state and persists this information to a local log file. Multiple Zookeeper460s can be used to support large Hadoop clusters, with a master server keeping the top-level servers in sync.

[0063] 4, pod1 412 and pod3 414 are inactive, and pod3 416 is active. In response to an instance becoming unavailable, the instance is switched from active to inactive.

[0064] Proxy layer 440 is capable of status checks 442 of pod 1 412, pod 2 414, and pod 3 416. Thus, user 450 can access an active instance without knowing which of pod 1 412, pod 2 414, and pod 3 416 is active or inactive. Proxy layer 440 determines which of pod 1 412, pod 2 414, and pod 3 416 is active.

[0065] When multiple instances of Apache Atlas 410 are started under the same Zookeeper 460, the proxy layer 440 automatically determines that Pod 3 416 is active and therefore selects Pod 3 to serve user requests. Other instances, such as Pod 1 412 and Pod 414, are automatically considered inactive. In response to an active instance being intentionally stopped or becoming unavailable due to a failure, one of the other instances is automatically selected as the active instance and started to serve user requests. An active instance is an instance that can properly respond to user requests. An active instance can create, delete, modify, or respond to queries for metadata objects. Zookeeper 460 assumes current leader election 462, i.e., the selection of an active instance (e.g., Pod 416), if Pod 412 and Pod 414 are down or inactive. The proxy layer 440 tracks the currently active instance and forwards requests 456, 458 to the active instance. Thus, Zookeeper can provide response 418 from the active leader, e.g., pod 3 416, to proxy layer 440. Proxy layer 440 then forwards response 444 to user 450 or response 446 to external job 452.

[0066] FIG. 5 shows a simple data lineage 500 provided by existing components.

[0067] 5, a first data set, User_Data 510, and a second data set, Order_Data 520, are combined in a data generation process 530 to create a third data set, Order_X_User_Data 540. A user can use the third data set, Order_X_User_Data 540. Data lineage identifies that the third data set, Order_X_User_Data 540, arose from the first data set, User_Data 510, and the second data set, Order_Data 520, and that the data generation process 530 was used to generate the third data set, Order_X_User_Data 540. Some existing products provide this type of data lineage. Some existing products also provide an SQL process to generate the third data set.

[0068] However, the problem with the data generation process 530 that transforms between the first data set, User_Data 510, and the second data set, Order_Data 520, is that it is more complex: the details of the complex transformation sequence cannot be reproduced as a single sequence.

[0069] FIG. 6 illustrates capturing discrete processes in detailed data lineage 600, according to at least one embodiment.

[0070] 6, the detailed data lineage 600 for joining User_Data 610 and Order_Data 620 is broken down to provide details of the process prior to the join process 630. Overall, the process is similar to when User_Data 610 and Order_Data 620 are transformed to produce Order_X_User_Data 640. However, the discrete internal processes that occur prior to producing the third data set, Order_X_User_Data 630, are captured.

[0071] 6, the detailed data lineage 600 is further decomposed at a lower level to represent intermediate data and processes, making the lineage more informative. Capturing discrete processes helps end users make appropriate decisions about how to use the data.

[0072] User_Data 610 is provided to User_Data_Modified_Generator 650 to create User_Data_Modified 642. As an example, User_Data 610 is modified according to a modification operation 654 of “select user_id,state,city,pincode,email,phone_number,split(name,'')(0) as f_name.split)name,'')(1) as l_name,toUpper(f_name),toUpper(l_name),todate(creation_timestamp) as creation_date,dated ff(now,creation_date) as system_age from user_data” to create User_Data_Modified 652. Those skilled in the art will recognize that the embodiments described herein are not meant to be limited to the modification operation 654 described herein, and that other or additional modifications may be made.

[0073] User_Data_Modified 652 is then filtered by User_Data_Filtered_Generator 660 to remove some irrelevant data and create User_Data_Filtered 662. For example, in Figure 6, User_Data_Modified 652 is filtered by User_Data_Filtered_Generator 660 according to a filter action 664 of "select user_id,state,city,pincode,email,phone_number,split(name,'')(0) as f_name.split(name,'')(1) as l_name,toUpper(f_name),toUpper(l_name),todate(creation_timestamp) as creation_date,dated ff(now,creation_date) as system_age from user_data_modified where phone_number is not null and system_age >= 365" to create User_Data_Filtered 662. However, those skilled in the art will recognize that the embodiments described herein are not meant to be limited to the filter operations 664 described herein, and that other or additional filter operations may be performed.

[0074] Next, Order_Data 620 is combined with User_Data_Filtered 662 to produce Order_X_User_Data 640. For example, User_Data_Filtered 662 is combined with Order_Data 620 using a join process 630 according to a join operation 632 of “select “from order_data a left join user_data_filtered b on a.user_id=b.user_id” to produce Order_X_User_Data 640. However, similar to the modify operation 654 and filter operation 664 described above, one skilled in the art will recognize that the embodiments described herein are not meant to be limited to the combine operation 632 described herein, and that other or additional join operations may be performed.

[0075] Capturing process details in detailed data lineage 600 allows users to determine what is happening internally, and is not limited to just some process, name, or basic sequence. Detailed data lineage 600 allows users to make confident decisions based on the process or sequence details.

[0076] FIG. 7 illustrates data pattern detection and optimization 700 according to at least one embodiment.

[0077] In FIG. 7, data services 710 include a data discovery interface 720 that allows users / applications to search 722 for data across a data catalog and data lineage 730 .

[0078] A data discovery interface 720 provides search queries 724 to the data catalog and data lineage 730. A background job 740 feeds data to a process 750 that automates data modeling and further data tagging and classification. A host artificial intelligence / machine learning (AI / ML) 760 runs in the background based on the data catalog and data lineage 730. A user 770 can use the AI / ML 760, for example, via a web user interface (UI) 772 or an application 780, to identify existing dataset patterns using suggestions of possible matches between datasets and the data catalog and data lineage 730. Thus, searching the data catalog and data lineage 730 involves the AI / ML crawling the data catalog and data lineage 730 and presenting suggestions of possible matches between similar datasets or similar sets of actions to perform to produce similar datasets.

[0079] In the data computation interface 790, a user can first select a source dataset, then define a set of operations / transformations, and run a job to perform the operations and generate a dataset 792. While defining these jobs, internally, the data computation interface 790, with inputs as source datasets and sets of transformations applied, presents the user with existing patterns in the system using the pattern detection capabilities provided by the data discovery interface 720. This prevents the user from creating duplicate sets of jobs and datasets.

[0080] In the data discovery interface 720, in response to a user 770 or application 780 searching for data with associated terms / classifications / tags, apart from the list of datasets, the results presented include a list of existing actions that can be applied to the item.

[0081] Similarly, any external applications 780 can use the pattern detection capabilities provided by AI / ML 760 to avoid duplication of datasets and computational jobs. Also, a background job runs that continuously feeds the data catalog and data lineage 730 into the hosted model 750. This helps create system-generated tags and classifications that can be used directly by users to enhance searches or internally by the data discovery interface 730 to discover similar datasets and transformation patterns.

[0082] FIG. 8 is a flowchart 800 of a method for providing near real-time data cataloging and large-scale data discovery with detailed lineage, according to at least one embodiment.

[0083] 8, the process begins S810 and data from one or more sources is received S810. Referring to FIG. 2, authentication and middleware interface 210 receives input from one or more sources 212, for example, via web UI 214 and / or onboarding interface 216.

[0084] The received data is written to the data storage system S814. Referring to Figure 2, the data computing interface 220 includes a data generation and transformation layer that writes or reads data to the data storage layer 230. The data governance (access control layer) 250 controls access to the data storage layer 230 by determining access rights to the data storage layer 230.

[0085] The received data is processed S818 in the data storage system using sensors in the data storage system. Referring to Figure 2, at "3", the data from sensors 238 is written to a database that classifies the data into new and changed data and captures change information that is pushed to data computing interface 220 and then data services interface 240. Sensors 238 detect changes in the data and then push the changes to data computing interface 220 and then data services interface 240. For example, a file is written to the database and sensor 238 sends the file to data computing interface 220. The metadata capture service retrieves these and creates data catalog 244 and data lineage 246.

[0086] The metadata and data definitions associated with the data are sent to the data service in near real time S822. Referring to Figure 2, at "5", the data computing interface 220 pushes metadata, creates data lineage, and audits data 280 to the data service interface 240. Data definitions may also be provided to the data service interface 240 by the data computing interface 220.

[0087] A data catalog is generated S826 at the data service based on the metadata associated with the data and the data definitions. Referring to Figure 2, data service interface 240 includes data discovery interface 242, which presents data catalog 244 and data lineage 246. In response to a file being created from an existing file, data lineage 246 is captured that identifies the file as being derived from a prior existing file. Metadata information 280 is presented to data service interface 240 by data computation interface 220 in a unified discovery format that is subsequently provided to the duplicate sensor.

[0088] Data lineage for the data is created in the data service S830. Referring to FIG. 2, at "6", duplicates in the data lineage and data catalog 281 are eliminated. New data paths that share the same job are added to the data lineage 246, and the data paths are associated under one data catalog 244 based on metadata and data definitions. The data catalog 244 presents a catalog related to information such as lineage, data path, owner, and data description in data discovery 242.

[0089] The data catalog and data lineage are presented to the data service's data discovery interface S834. Referring to FIG. 2, a user can search and discover 270 data of interest across the data by accessing the data discovery interface 242 in the data service interface 240. At "4," the data computation interface 220 retrieves the metadata change event 228. Data lineage 246 is created to present detailed, granular lineage information, including details such as data classification and business topology tagged in the metadata, for use in performing complex data transformations.

[0090] Input is received via the data discovery interface S838 to query metadata associated with the data in the data storage system. Referring to Figure 2, data lineage and data audit 280 is pushed from data computing interface 220 to data services interface 240. Data services interface 240 receives input to query metadata associated with the data in the data storage system. Tag data and created classifications and categorizations 271 are associated with the data.

[0091] A data set in the data storage system is identified S842 by the query interface based on the input, the metadata, and the data definitions. Referring to Figure 2, when a query is generated by the data query interface 260, a data set is identified by the data query interface 260 in the data storage layer 230 based on the input to the data discovery interface 242 of the data services interface 240 and the metadata and data definitions fetched from the data services interface 240.

[0092] Access to the identified data set is requested S846 at the data storage system. Referring to Figure 2, the data governance (access control layer) 250 controls access to the data storage layer 230 by requesting access rights to the data storage layer 230.

[0093] In response to the access permissions for the identified dataset, the query interface generates a query to retrieve the identified dataset based on the input, the metadata, and the data definition S850. Referring to FIG. 2 , in response to the access permissions for the identified dataset, the query interface 260 generates a query to retrieve the identified dataset based on the input, the metadata, and the data definition. The data storage layer 230 enhances data discovery availability for data in the data storage system in response to the query and includes a proxy layer that processes search requests for and responses from multiple applications under the control of a central coordinator in the data storage layer 230, which identifies active readers from the applications and enables the proxy layer to forward requests to and receive responses from the active readers. Data is then fetched directly from the data sources in the data storage layer 230.

[0094] The process then terminates S860.

[0095] At least one embodiment of a method for providing large-scale data discovery with near real-time data cataloging and detailed lineage includes receiving data from one or more sources, writing the received data to a data storage system, using sensors at the data storage system to process the data received at the data storage system, sending metadata and data definitions associated with the data to a data service in near real time, generating a data catalog at the data service based on the metadata and data definitions associated with the data, creating data lineage for the data at the data service, and presenting the data catalog and data lineage in a data discovery interface.

[0096] FIG. 9 is a high-level functional block diagram of a processor-based system 900 according to at least one embodiment.

[0097] In at least one embodiment, the processing circuit 900 provides large-scale data discovery with near real-time data cataloging and detailed lineage. The processing circuit 900 implements the large-scale data discovery with near real-time data cataloging and detailed lineage using a processor 902. The processing circuit 900 also includes a non-transitory computer-readable storage medium 904 used to implement a data platform for providing large-scale data discovery with near real-time data cataloging and detailed lineage. The non-transitory computer-readable storage medium 904, among other things, stores instructions 906, i.e., encoded with computer program code, that are executed by the processor 902 to cause the processor 902 to perform operations for providing large-scale data discovery with near real-time data cataloging and detailed lineage. Execution of the instructions 906 by the processor 902 represents (at least in part) an application that implements at least a portion of a methodology described herein (hereinafter, the described process and / or methodology) in accordance with one or more embodiments.

[0098] The processor 902 is electrically coupled to a non-transitory computer-readable storage medium 904 via a bus 908. The processor 902 is electrically coupled to an input / output (I / O) interface 910 by the bus 908. A network interface 912 is also electrically connected to the processor 902 via the bus 908. The network interface 912 is connected to a network 914 such that the processor 902 and the non-transitory computer-readable storage medium 904 connect to external elements via the network 914. The processor 902 is configured to execute instructions 906 encoded in the non-transitory computer-readable storage medium 904 to enable the processing circuit 900 to perform at least a portion of a process and / or method. In one or more embodiments, the processor 902 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or other suitable processing unit.

[0099] Processing circuit 900 includes an I / O interface 910. I / O interface 910 is coupled to external circuitry. In one or more embodiments, I / O interface 910 includes a keyboard, keypad, mouse, trackball, trackpad, touch screen, and / or cursor direction keys for communicating information and commands to processor 902.

[0100] The processing circuit 900 also includes a network interface 912 coupled to the processor 902. The network interface 912 enables the processing circuit 900 to communicate with a network 914 to which one or more other computer systems are connected. The network interface 912 includes a wireless network interface such as Bluetooth, Wi-Fi, Worldwide Interoperability for Microwave Access (WiMAX), General Packet Radio Service (GPRS), or Wideband Code Division Multiple Access (WCDMA), or a wired network interface such as Ethernet, Universal Serial Bus (USB), or Institute of Electrical and Electronics Engineers (IEEE) 864.

[0101] The processing circuit 900 is configured to receive information via an I / O interface 910. The information received via the I / O interface 910 includes one or more of instructions, data, design rules, libraries of cells, and / or other parameters for processing by the processor 902. The information is transferred to the processor 902 via the bus 908. The processing circuit 900 is configured to receive information related to a user interface (UI) via the I / O interface 910. The information is stored in the non-transitory computer-readable storage medium 904 as a UI 922.

[0102] In one or more embodiments, one or more non-transitory computer-readable storage media 904 store instructions 906 (in compressed or uncompressed format) that can be used to program a computer, processor, or other electronic device to perform the processes or methods described herein. The one or more non-transitory computer-readable storage media 904 include one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, etc.

[0103] For example, the non-transitory computer-readable storage medium 904 may include, but is not limited to, a hard drive, a floppy diskette, an optical disk, a read-only memory (ROM), a random-access memory (RAM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a magnetic or optical card, a solid-state memory device, or any other type of physical medium suitable for storing electronic instructions. In one or more embodiments using an optical disk, the one or more non-transitory computer-readable storage media 904 include a compact disk-read only memory (CD-ROM), a compact disk-read / write (CD-R / W), and / or a digital video disc (DVD).

[0104] In one or more embodiments, the non-transitory computer-readable storage medium 904 stores instructions 906 configured to cause the processor 902 to perform at least a portion of the processes and / or methods for implementing the data platform described herein. In one or more embodiments, the non-transitory computer-readable storage medium 904 also stores information, such as algorithms, that facilitate the execution of at least a portion of the processes and / or methods for implementing the data platform described herein. Thus, in at least one embodiment, the processor 902 executes the instructions 906 on the non-transitory computer-readable medium to implement an authentication and middleware interface 926 for receiving input from one or more sources, for example, via a user interface 920, such as a web UI and / or an onboarding interface. The processor further executes the instructions 906 on the non-transitory computer-readable medium to implement data governance 921, a data storage layer 922, a data computation interface 923, a data discovery interface 924, and a query interface 925. The processor 902 executes instructions 906 on a non-transitory computer-readable medium to implement a user interface 932 on a display 930 to provide near-real-time data cataloging and large-scale data discovery with detailed lineage. The user interface 932 can display data as managed in the data platform 900 by the processor 902. The user interface 932 includes a data computation interface 940, a data services interface 950, and a query interface 960. The data computation interface 940 includes a data generation and transformation layer that writes data to or reads data from the data storage layer. The data governance (access control layer) controls access to the data storage layer 922 by determining access rights to the data storage layer 922. New data and changes are captured by the data storage layer 922 and pushed to the data computation interface 940 and then to the data services interface 950.For example, files are written to a database, and sensors 972 in data storage system 970 send files to data computing interface 940. A metadata capture service retrieves these and creates a data catalog 954 and data lineage 956 in data service interface 950. Data computing interface 940 pushes metadata, creates data lineage 956, and performs data audits. This information is pushed to data service interface 950 in near real time. Data definitions can also be provided to data service interface 950 by data computing interface 940. Data service interface 950 includes a data discovery interface 952 that presents data catalog 954 and data lineage 956. In response to a file being created from an existing file, data lineage 956 is captured, identifying the file as derived from a prior existing file. Metadata information is presented to data service interface 950 by data computing interface 940 in a unified discovery format that is then provided to duplicate sensors. Duplicates in the data lineage 956 and data catalog 954 are determined and removed. New data paths that share the same job are added to the data lineage 956, and data paths are associated under one data catalog 954 based on metadata and data definitions. The data catalog 954 presents related catalogs of information such as lineage, data paths, owners, and data descriptions in a data discovery interface 952. Users can search and discover data of interest across the data by accessing the data discovery interface 952 in the data services interface 950. The data computation interface 940 captures metadata change events. Data lineage 956 is created to present detailed, granular lineage information, including details such as data classification and business topology tagged in metadata, for use in performing complex data transformations.Data lineage 956 and data audits are pushed from the data computation interface 940 to the data services interface 950. The data services interface 950 receives input at a data discovery interface 952 to query metadata associated with data in a data storage system 970. Tag data, classification data, and categorization data are associated with the data. When a query is generated by the data query interface 960, datasets are identified by the data query interface 960 in the data storage layer 922 based on the input to the data discovery interface 952 of the data services interface 950 and the metadata and data definitions fetched from the data services interface 950. The data governance (access control layer) controls access to the data storage layer 922 by requesting access rights to the data storage layer 922. In response to access permissions for the identified datasets, the data query interface 960 generates queries to retrieve the identified datasets based on the input, metadata, and data definitions. The data storage layer 922 provides high availability for data discovery of data in the data storage system 970 in response to queries, and includes a proxy layer that processes search requests for and responses from multiple applications under the control of a central coordinator in the data storage layer 922, which identifies an active leader from the applications and enables the proxy layer to forward requests to and receive responses from the active leader. The data is then fetched directly from the data storage system 970 by the data storage layer 922.

[0105] Accordingly, the embodiments described herein provide certain advantages, including one or more of: providing a one-stop solution for the data usage lifecycle; providing near-real-time discoverability of data to end users with minimal manual intervention; enabling informed business decisions through low-level lineage including detailed procedures and other metadata details, including schemas, related to data transformations; providing ease of use through automated creation of a data catalog and integration with the query layer; or provisioning appropriate data semantics and accountability. The pattern detection feature helps any user discover / search for any data based on the type of action (or set of actions) that led to the generation of that particular data. This knowledge helps users / applications reduce their costs and efforts by avoiding duplication of datasets and the actions used to generate them.

[0106] One aspect of the present specification is directed to a method [1] for providing a data platform, the method comprising: receiving data from one or more sources; writing the received data to one or more data storage systems; using sensors at the data storage systems to process the received data at the one or more data storage systems; transmitting metadata and data definitions associated with the data to a data service in near real time; generating, at the data service, a data catalog based on the metadata and data definitions associated with the data; creating, at the data service, data lineage for the data; and presenting the data catalog and data lineage in a data discovery interface.

[0107] The method described in [1] further includes receiving input via a data discovery interface for querying metadata associated with data in the data storage system, and identifying a dataset in the data storage system based on the input, the metadata, and the data definition.

[0108] The method according to [1] to [2] further comprises the steps of requesting access to the identified dataset in the data storage, and generating a query to retrieve the identified dataset based on the input, the metadata and the data definition in response to access permissions for the identified dataset.

[0109] The methods described in [1] to [3] further include increasing the availability of data discovery of data in a data storage system in response to a query by providing a proxy layer that processes search requests to and responses from multiple applications under the control of a central coordinator, wherein the central coordinator identifies an active leader from the applications and enables the proxy layer to forward requests to and receive responses from the active leader.

[0110] The method according to [1] to [4] further includes the steps of creating, in the data service, a data audit associated with the data, and obtaining, by the data computation interface, change events representing the data for one or more sources in the data storage system.

[0111] In the methods described in [1] to [5], creating data lineage includes identifying discrete processes in a data flow associated with the data; using artificial intelligence to perform pattern recognition on metadata associated with the data to identify and eliminate duplicates in the data lineage and data catalog associated with the data by eliminating duplicates in the data lineage and data catalog; and presenting detailed, granular data lineage and data catalog for use in performing complex data transformations.

[0112] In the method according to any one of [1] to [6], writing the received data to the data storage system comprises determining access rights to the data storage system for writing the data to the data storage system.

[0113] One aspect of the present specification is directed to a data platform [8] including a memory storing computer-readable instructions and a processor coupled to the memory, the processor configured to execute the computer-readable instructions to perform the following operations: receive data from one or more sources; write the received data to a data storage system; process the data received at the data storage system using sensors at the data storage system; transmit metadata and data definitions associated with the data to a data service in near real time; generate, at the data service, a data catalog based on the metadata and data definitions associated with the data; create, at the data service, data lineage for the data; and present the data catalog and data lineage in a data discovery interface.

[0114] In the data platform described in [8], the processor is further configured to receive input via a data discovery interface for querying metadata associated with data in a data storage system, and identify a dataset in the data storage system based on the input, the metadata, and the data definition.

[0115] In the data platform described in [8] to [9], the processor is further configured to request access to the identified dataset in the data storage system and generate a query to retrieve the identified dataset based on the input, the metadata, and the data definition in accordance with access permissions for the identified dataset.

[0116] In the data platform described in [8] to

[10] , the processor is further configured to increase availability of data discovery of data in the data storage system in response to queries by providing a proxy layer that processes search requests to and responses from multiple applications under the control of a central coordinator, the central coordinator identifying an active reader from the applications and enabling the proxy layer to forward requests to and receive responses from the active reader.

[0117] In the data platform described in [8] to

[11] , the processor is further configured to create a data audit associated with the data in the data service and obtain change events representing the data for one or more sources in the data storage system.

[0118] In the data platform described in [8] to

[12] , the processor is further configured to create data lineage by identifying discrete processes in a data flow associated with the data, performing pattern recognition on metadata associated with the data using artificial intelligence to identify duplicates in a data lineage and data catalog associated with the data, eliminating duplicates in the data lineage and data catalog associated with the data, and presenting detailed, granular data lineage and data catalog for use in performing complex data transformations.

[0119] In the data platform described in [8] to

[13] , the processor is further configured to write the received data to the data storage system by determining access rights to the data storage system to write the data to the data storage system.

[0120] One aspect of the present specification is directed to a non-transitory computer-readable medium

[15] having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the following operations: receive data from one or more sources; write the received data to a data storage system; use sensors at the data storage system to process the data received at the data storage system; transmit metadata and a data definition associated with the data to a data service in near real time; at the data service, generate a data catalog based on the metadata and the data definition associated with the data; at the data service, create data lineage for the data; and present the data catalog and data lineage in a data discovery interface.

[0121] The non-transitory computer-readable medium described in

[15] further includes receiving input via a data discovery interface for querying metadata associated with data in a data storage system; identifying a dataset in the data storage based on the input, the metadata, and the data definition; requesting access to the identified dataset in the data storage system; and generating a query to retrieve the identified dataset based on the input, the metadata, and the data definition according to access permissions for the identified dataset.

[0122] The non-transitory computer-readable medium described in

[15] to

[16] further includes increasing availability of data discovery of data in a data storage system in response to a query by providing a proxy layer that processes search requests for and responses from multiple applications under the control of a central coordinator, wherein the central coordinator identifies an active reader from the applications and enables the proxy layer to forward requests to and receive responses from the active reader.

[0123] The non-transitory computer-readable medium described in

[15] to

[17] further includes, in a data service, creating a data audit associated with the data, and capturing change events representing the data for one or more sources in a data storage system.

[0124]

[15] to

[18] , in the non-transitory computer-readable medium, creating data lineage includes identifying discrete processes in a data flow associated with the data, using artificial intelligence to perform pattern recognition on metadata associated with the data to identify duplicates in the data lineage and data catalog associated with the data, eliminating duplicates in the data lineage and data catalog associated with the data, and presenting detailed, granular data lineage and data catalog for use in performing complex data transformations.

[0125] In the non-transitory computer-readable medium described in

[15] to

[19] , writing the received data to the data storage system includes determining access rights to the data storage system to write the data to the data storage system.

[0126] Separate instances of these programs may run on or be distributed on any number of separate computer systems. Thus, although particular procedures may have been described as being performed by particular devices, software programs, processes, or entities, this is not necessarily the case. Various alternative implementations will be appreciated by those skilled in the art.

[0127] Furthermore, those skilled in the art will readily recognize that the techniques described above can be utilized in a variety of devices, environments, and contexts. Although the embodiments have been described in language specific to structural features or methodological acts, the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. 1. A method for providing a data platform, comprising: receiving data from one or more sources; writing the received data to one or more data storage systems; processing the received data at the data storage systems using sensors at the one or more data storage systems; transmitting metadata and data definitions associated with the data received from the one or more data storage systems to a data service in near real time; generating a data catalog based on the metadata and data definitions associated with the data in the data service; creating data lineage for the data in the data service; and presenting the data catalog and the data lineage in a data discovery interface. method.

2. receiving input via the data discovery interface for querying the metadata associated with the data in the data storage system; and further comprising identifying a data set in the data storage system based on the input, the metadata, and the data definition. The method of claim 1.

3. requesting access to the identified data set in the data storage system; and and generating a query to retrieve the identified dataset based on the input, the metadata, and the data definition in response to access permissions for the identified dataset. The method of claim 2.

4. providing a proxy layer that processes search requests to and responses from multiple applications under the control of a central coordinator, thereby increasing availability of data discovery of the data in the data storage system in response to queries; the central coordinator identifies an active leader from the application and enables the proxy layer to forward requests to and receive responses from the active leader; The method of claim 1.

5. creating, in the data service, a data audit associated with the data; and and acquiring, by the data computing interface, change events representing the data for the one or more sources in the data storage system. The method of claim 1.

6. The creation of the data lineage includes: identifying a discrete process within the data flow associated with said data; removing the duplications in the data lineage and data catalog associated with the data by performing pattern recognition on the metadata associated with the data using artificial intelligence to identify duplications in the data lineage and data catalog associated with the data and remove the duplications in the data lineage and data catalog; and presenting detailed, granular data lineage and data catalogs for use in performing complex data transformations, including; The method of claim 1.

7. writing the received data to the data storage system includes determining access rights to the data storage system to write the data to the data storage system; The method of claim 1.

8. A data platform, a memory storing computer readable instructions; a processor coupled to the memory; The processor is configured to execute the computer-readable instructions to perform operations, including: receiving data from one or more sources; writing the received data to a data storage system; processing the received data at the data storage system using a sensor at the data storage system; transmitting metadata and data definitions associated with said data to a data service in near real time; generating a data catalog based on the metadata and data definitions associated with the data in the data service; creating data lineage for the data in the data service; and presenting the data catalog and the data lineage in a data discovery interface. Data platform.

9. The processor: receiving input via the data discovery interface for querying the metadata associated with the data in the data storage system; and and further configured to identify a dataset in the data storage system based on the input, the metadata, and the data definition. The data platform according to claim 8 .

10. The processor: requesting access to the identified data set in the data storage system; and and further configured to generate a query to retrieve the identified dataset based on the input, the metadata, and the data definition in response to access permissions for the identified dataset. The data platform according to claim 9 .

11. the processor is further configured to increase availability of data discovery of the data in the data storage system in response to a query by providing a proxy layer that processes search requests for and responses from multiple applications under control of a central coordinator; the central coordinator identifies an active leader from the application and enables the proxy layer to forward requests to and receive responses from the active leader; The data platform according to claim 8 .

12. The processor creates a data audit associated with the data at the data service; and and further configured to: obtain change events representing the data for the one or more sources at the data storage system. The data platform according to claim 8 .

13. The processor may further process the creation of the data lineage by: identifying a discrete process within the data flow associated with said data; removing the duplications in the data lineage and data catalog associated with the data by performing pattern recognition on the metadata associated with the data using artificial intelligence to identify duplications in the data lineage and data catalog associated with the data and remove the duplications in the data lineage and data catalog; and creating the data lineage by presenting detailed, granular data lineage and a data catalog for use in performing complex data transformations. The data platform according to claim 8 .

14. the processor is further configured to write the received data to the data storage system by determining access rights to the data storage system for writing the data to the data storage system. The data platform according to claim 8 .

15. A non-transitory computer-readable medium having computer-readable instructions stored thereon, comprising: The computer-readable instructions, when executed by a processor, are configured to cause the processor to perform operations, including: receiving data from one or more sources; writing the received data to a data storage system; processing the received data at the data storage system using a sensor at the data storage system; transmitting metadata and data definitions associated with said data to a data service in near real time; generating a data catalog based on the metadata and data definitions associated with the data in the data service; creating data lineage for the data in the data service; and presenting the data catalog and the data lineage in a data discovery interface. Non-transitory computer-readable medium.

16. receiving input via the data discovery interface for querying the metadata associated with the data in the data storage system; identifying a data set in the data storage system based on the input, the metadata, and the data definition; requesting access to the identified data set in the data storage system; and and generating a query to retrieve the identified dataset based on the input, the metadata, and the data definition in response to access permissions for the identified dataset.

16. The non-transitory computer-readable medium of claim 15.

17. providing a proxy layer that processes search requests to and responses from multiple applications under the control of a central coordinator, thereby increasing availability of data discovery of the data in the data storage system in response to queries; the central coordinator identifies an active leader from the application and enables the proxy layer to forward requests to and receive responses from the active leader; 17. The non-transitory computer-readable medium of claim 16.

18. creating, in the data service, a data audit associated with the data; and obtaining change events representing the data for the one or more sources at the data storage system; 16. The non-transitory computer-readable medium of claim 15.

19. The creation of the data lineage includes: identifying a discrete process within the data flow associated with said data; removing the duplications in the data lineage and data catalog associated with the data by performing pattern recognition on the metadata associated with the data using artificial intelligence to identify duplications in the data lineage and data catalog associated with the data and remove the duplications in the data lineage and data catalog; and presenting detailed, granular data lineage and data catalogs for use in performing complex data transformations, including; 16. The non-transitory computer-readable medium of claim 15.

20. writing the received data to the data storage system includes determining access rights to the data storage system to write the data to the data storage system; 16. The non-transitory computer-readable medium of claim 15.

Images