Metadata on incremental shared tables

Abstract

In an example embodiment, additional metadata is attached to an increment table in the form of a common pattern symbol (CSN) entity via increment sharing. When a request for sharing a range of delta tables is received from a client, where the range includes a delta table to which additional metadata is attached, the latest CSN entity of each of the delta tables in the range is aggregated and returned to the client.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 703,519, filed October 4, 2024, entitled “METADATA ON DELTA SHARING TABLES,” the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to data lake storage, and in particular to metadata on delta sharing tables of a data lake. BACKGROUND

[0004] A data lake is a single, centralized repository in which an organization can store data in structured, unstructured, and semi-structured formats. This allows organizations to more quickly and easily store, access, and analyze a wide variety of data in a single location. Unlike a database, data stored in a data lake does not need to fit into a particular structure format. Instead, data can be stored in its original or native format, often as a file or binary large object (BLOB). SUMMARY

[0005] According to at least one embodiment of the present disclosure, a system is provided, comprising: at least one hardware processor; and a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: generating a first piece of metadata for a delta table in a data lake, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between multiple versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving a request from a client for sharing of a range of the delta table, the range including the delta table; aggregating a latest common schema file for each delta table in the range into an aggregated common schema file; and sharing the aggregated common schema file with the client.

[0006] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema file for each delta table in the range.

[0007] According to at least one embodiment of the present disclosure, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file in the delta table in the range.

[0008] According to at least one embodiment of the present disclosure, wherein the aggregating is asynchronously triggered.

[0009] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the operations further comprise restoring consistency of the aggregated common schema file using the aggregation timestamp.

[0010] According to at least one embodiment of the present disclosure, wherein the first piece of metadata is used to train a machine learning model.

[0011] According to at least one embodiment of the present disclosure, wherein the sharing is performed via a delta sharing application program interface (API).

[0012] According to at least one embodiment of the present disclosure, a method is provided, comprising: generating, for a delta table in a data lake, a first piece of metadata, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between a plurality of versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving, from a client, a request for sharing of a scope of the delta table, the scope including the delta table; aggregating, into an aggregated common schema file, a latest common schema file of each delta table in the scope; and sharing, with the client, the aggregated common schema file.

[0013] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema file of each delta table in the scope.

[0014] According to at least one embodiment of the present disclosure, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file in the delta table in the scope.

[0015] According to at least one embodiment of the present disclosure, wherein the aggregating is asynchronously triggered.

[0016] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the method further comprises restoring consistency of the aggregated common schema file using the aggregation timestamp.

[0017] According to at least one embodiment of the present disclosure, wherein the first piece of metadata is used to train a machine learning model.

[0018] According to at least one embodiment of the present disclosure, wherein the sharing is performed via a delta sharing application program interface (API).

[0019] According to at least one embodiment of the present disclosure, there is provided a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: generating a first piece of metadata for a delta table in a data lake, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between a plurality of versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving a request from a client for a share of a range of the delta table, the range including the delta table; aggregating a latest common schema file for each delta table in the range into an aggregated common schema file; and sharing the aggregated common schema file with the client.

[0020] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema file for each delta table in the range.

[0021] According to at least one embodiment of the present disclosure, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file in the delta table in the range.

[0022] According to at least one embodiment of the present disclosure, wherein the aggregating is asynchronously triggered.

[0023] According to at least one embodiment of the present disclosure, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the operations further comprise restoring a consistency of the aggregated common schema file using the aggregation timestamp.

[0024] According to at least one embodiment of the present disclosure, wherein the first piece of metadata is used to train a machine learning model. BRIEF DESCRIPTION OF DRAWINGS

[0025] The disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that

[0026] Figure 1 is a block diagram illustrating a system for HDL file management according to example embodiments.

[0027] Figure 2 is a sequence diagram illustrating binding of a CSN model with a delta share table according to example embodiments.

[0028] Figure 3 is a flowchart illustrating a method for attaching metadata to a delta table according to an example embodiment.

[0029] Figure 4 is a block diagram illustrating a software architecture, which can be installed on any one or more of the above devices.

[0030] Figure 5 FIG. 1 1 shows a diagrammatic representation of machine in the form of a computer system within which a set of instructions can be executed for causing the machine to perform any one or more of the methodologies discussed herein, according to an example embodiment. DETAILED DESCRIPTION

[0031] The following description discusses illustrative systems, methods, techniques, instruction sequences, and computer machine program products. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various example embodiments of the present subject matter. It will be evident, however, to those skilled in the art, that embodiments of the present subject matter can be practiced without some or all of these specific details.

[0032] Technical problems are encountered with the use of data lakes, particularly with respect to the use of delta tables. Delta tables work by combining elements of data lakes and data warehouses to provide a more reliable solution for data processing than a data lake alone. Specifically, a delta table is stored in a specialized format built on top of an existing data storage system. It adds extra metadata and transaction logs to data lake files.

[0033] The transaction logs record all changes (e.g., writes, updates, deletes) made to the delta table, which helps ensure that all operations complete successfully or roll back if there is a problem, and also allows for querying historical versions of data and managing concurrent reads and permissions.

[0034] Delta tables can also support a delta sharing protocol that enables secure sharing of the delta table.

[0035] However, the metadata model of the delta sharing and delta table does not cover all relevant metadata. It is missing items such as strong table schema definition, primary key specification, relationships between tables, i18next of metadata, and business semantics. All of these items are useful for downstream processing in a cloud storage environment. These downstream processing tasks can involve, for example, using the data in machine learning models, in training those machine learning models, or in making predictions using the machine learning models (or both).

[0036] It is not feasible to change the metadata model to include such additional items, and so an alternative solution is needed to address the lack of strong table schema definition, primary key specification, relationships between tables, i18next of metadata, and business semantics in the delta table metadata.

[0037] In example embodiments, the additional metadata is attached to the delta table in the form of common schema notation (CSN) entities via delta sharing in a common schema notation (CSN) entity. CSN is a standard format used primarily in the context of core data services (CDS) to define and document the structure of data models. CDS is a framework for defining database schemas and CSN is a way to represent these schemas in a standardized, machine-readable format.

[0038] Data lake files can sometimes be stored in in-memory data stores such as HANA™ from SAP, Walldorf, Germany. Thus, data lake files are sometimes referred to as HDL files.

[0039] The HDL files can then expose the aggregated CSN metadata for discovery of delta shares and delta tables with rich semantics from within the unified customer landscape (URL) by, for example, a discovery protocol such as the Open Resource Discovery (ORD) protocol.

[0040] Since the delta table itself provides versioning of the schema / data as well as schema evolution, in example embodiments, this versioning and evolution is extended to the CSN metadata definition. When a CSN document is provided for a table, it is thus allowed for the user to specify the starting delta table version from which the CSN document is valid. This applies to all table versions until a new update to the same table provides different CSN entities for a higher version of the delta table. Thus, for example, consider the following table:

[0041]

[0042] This example represents a scenario where the table has three CSN documents attached to it: 0, 1, and 2, and they apply to the following table version ranges respectively: 0-22, 23-44, and 45-.

[0043] Figure 1is a block diagram illustrating a system 100 for HDL file management according to example embodiments. A file storage component 102 stores the HDL files themselves. The file storage component 102 can be in the form of, for example, a Web Hadoop Distributed File System (WebHDFS) repository or database. WebHDFS is a Representational State Transfer (REST) Application Program Interface (API) for accessing HDFS files. It provides a web-based interface to interact with HDFS, allowing file storage and retrieval operations over Hypertext Transfer Protocol (HTTP). Thus, the file storage component 102 can have one or more file storage APIs 104, such as WebHDFS, which can be used to access, upload, download, and manage files stored in the file storage component 102.

[0044] A catalog 106 manages metadata related to the HDL files, such as table definitions, schema information, and file properties. This facilitates efficient organization and querying of the metadata. The catalog 106 includes tables that describe the structure, relationships, and properties of the HDL files. One or more catalog APIs 108 can be used to perform querying and management of this metadata.

[0045] A delta sharing repository 110 stores information about delta sharing. Delta sharing involves sharing incremental changes (deltas) between different versions of a delta table in an HDL file. The delta sharing repository 110 tracks these changes and facilitates efficient sharing. This can involve storing delta files or logs that capture modifications, additions, or deletions that occur between different versions of an HDL file. The delta sharing repository 110 has one or more delta sharing APIs 112 to perform these tasks.

[0046] A cache and orchestration layer 114 manages interactions with the system 100 and the one or more file storage APIs 104, the one or more catalog APIs 108, and the one or more delta sharing APIs 112. More specifically, it can cache frequently accessed HDL files, metadata, and delta tables, and handle workflows, process automation, and ensure smooth flow of data between the file storage component 102, the catalog 106, and the delta sharing repository 110.

[0047] The storage abstraction layer 116 provides a consistent API for interacting with different types of storage systems and hides the details of where and how data is stored, allowing applications to interact with data in a uniform manner regardless of the underlying storage technology. Thus, for example, external hyperscalers 118A, 118B, 118C can interact with the file storage component 102, the catalog 106, and the delta share repository 110 without knowing the details of how these components operate. The storage abstraction layer 116 abstracts the one or more file storage APIs 104, the one or more catalog APIs 108, and the one or more delta share APIs 112.

[0048] The authentication component 120 ensures that only authorized users and systems can access and modify HDL files or metadata.

[0049] In example embodiments, the one or more file storage APIs 104, the one or more catalog APIs 108, and the one or more delta share APIs 112 are modified to accommodate the use of CSN entities as additional metadata by providing the ability to bind CSN models to existing delta share tables.

[0050] Specifically, a new endpoint can be introduced in the catalog 106 to allow users to bind CSN models to existing delta share tables. This means that a user creates a share table in the catalog and then attaches CSN metadata to the CSN endpoint of the share table with a subsequent request. Figure 2 is a sequence diagram illustrating the binding of a CSN model to a delta share table according to example embodiments. Specifically, at operation 202, a producer 200 creates a delta table in the file storage component 102 and, at operation 204, generates a notification indicating that the delta table has been created. The delta table is then added to the catalog 106 at operation 206, with a notification being generated at operation 208 indicating that the delta table has been added to the catalog 106.

[0051] At operation 210, the producer 200 places the schema table in the catalog 106 and, at operation 212, generates a notification indicating that the schema table has been created in the catalog 106. At operation 214, the producer 200 puts the CSN document into the catalog 106 and, at operation 216, generates a notification indicating that the CSN document has been created and bound to the delta table.

[0052] To bind a given table ( <share> 、 <schema> 、

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0142]

[0143]

[0144]

[0145]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

[0205]

[0206]

[0207]

[0208]

[0209]

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216]

[0217]

[0219]

[0220]

[0221]

[0222]

[0223]

[0225]

[0226]

[0227]

[0228]

[0229]

[0231]

[0232]

[0233]

[0234]

[0235]

[0236]

[0237]

[0238]

[0239]

[0240]

[0241]

[0242]

[0243]

[0244]

[0245]

[0246]

[0247]

[0248]

[0249]

[0250]

[0251]

[0252]

[0253]

[0254]

[0255]

[0256]

[0257]

[0258]

[0259]

[0260]

[0261]

[0262]

[0263]

[0265]

[0266]

[0267]

[0268]

[0269]

[0270]

[0271]

[0272] <share-oid> / @@csn / aggregate-event- <timestamp>Check the persistence section for more details.

[0273] 2. Handling request operations

[0274] That is, in case of creation / update of a shared table CSN, store the CSN provided for the shared table, in case of deletion of a shared table CSN, delete the shared table CSN, or in case of deletion of a shared table, delete the shared table.

[0275] 3. Asynchronous aggregation of shared CSNs

[0276] With the distributed executor, the aggregation task should be submitted to the node owning the share (i.e. the IMDG node holding the data partition containing the shared key). The same shared aggregation task should be synchronized.

[0277] The task should start by checking if an existing aggregation event marker exists, otherwise, exit the execution as there is nothing to do since the aggregation might have been overwritten / processed by a previous task (batch event processing).

[0278] The resulting aggregated CSN document, as well as the timestamp when the task started, should be stored in the shared metadata directory (... <share-oid>in a dedicated file under / @@csn.json. After storing the CSN, the obsolete event markers, i.e. all event markers created before starting the execution of the aggregation task, can be deleted. Note that this can be checked based on the timestamp that is part of the marker file name.

[0279] Given that the aggregation happens during the write operation and is stored in a file, the get shared CSN API can simply retrieve the shared CSN from this file.

[0280] To be able to recover from scenarios where the aggregation task is lost and there are unprocessed aggregation event markers, in case the last aggregation time (persisted together with the aggregated CSN document) is too old, a new asynchronous task can be submitted.

[0281] As for the persistence of the metadata of the CSN entity, the following file structure represents one way in which the directory 106 can persist metadata for a share and a shared table:

[0282]

[0283] The aggregated CSN of a share can be stored in its dedicated JSON file under the share metadata prefix. It should contain not only the aggregated CSN document but also the last aggregation timestamp, so that in case the current state is considered too old, an operation to recover the consistency of the shared CSN can happen periodically.

[0284] {

[0285] "shareCsn": <share-agg-csn>,

[0286] "lastAggregationTime": <timestamp>

[0287] }

[0288] The poly-event marker should be kept under a dedicated directory (@@csn) under the shared metadata prefix as well.

[0289] The CSN metadata of a shared table will be stored in the corresponding shared table JSON file together with the already existing table metadata. An additional field called csnEntities will be introduced and will contain the contents of the CSN documents. The default CSN document of a table, if any, will be saved under the default subfield, while the CSNs created for specific table versions will be saved separately under a version map whose keys indicate the table start version of the corresponding CSN. The following example illustrates the contents of a metadata JSON file of a shared table containing a default CSN document and three version CSN documents.

[0290] {

[0291] "location": "hdlfs: / / fe3bd16d-0d10-4695-abba-93d724673d64.files.hdl.prod-us30.hanaclou

[0292] "partitions": [

[0293] {

[0294] "name": "date",

[0295] "op": "EQUAL",

[0296] "value": "2021-04-29T00:00:00.000Z"

[0297] }

[0298] ],

[0299] "csnEntities": {

[0300] "default": <default-csn-doc>,

[0301] "version": {

[0302] "0": <csn-doc-valid-from-delta-table-version-0> ,

[0303] "5": <csn-doc-valid-from-delta-table-version-5> ,

[0304] "12": <csn-doc-valid-from-delta-table-version-12>

[0305] }

[0306] }

[0307] }

[0308] The first versioned CSN with key 0 would be valid for table versions between 0 and 4. The second versioned CSN with key 5 would be valid for table versions between 5 and 11. The last version CSN with key 12 would be valid for any table version >= 12.

[0309] Storing the CSN documents into a shared table file avoids the need for additional storage operations, which is desirable for performance reasons. For example, if each CSN were stored in a separate file, the prefix containing the CSN files would have to be listed to identify the CSN file that best matches the user request.

[0310] Because the CSN metadata only describes the table schema structure, each document tends to contain data in the order of a few kilobytes. Therefore, this should not cause scalability issues in the sense that the shared table file will not contain very large amounts of data.

[0311] A caching layer can be introduced to keep the metadata of the shared and shared tables in memory and allow faster queries and also reduce the amount of storage operations.

[0312] In another example embodiment, access control can be provided for the CSN metadata separately from the underlying delta table. For example, this allows a client to be granted permission to the CSN metadata associated with a delta table without being granted permission to the delta table itself, and vice versa.

[0313] Figure 3 is a flowchart illustrating a method 300 for attaching metadata to a delta table, according to an example embodiment. At operation 302, a first piece of metadata for a delta table in a data lake is generated. The data lake is a storage that contains structured, unstructured, and semi-structured data. The delta table describes differences between multiple versions of a data class file stored in the data lake. The first piece of metadata has a first metadata type that is not supported by a format of the delta table.

[0314] At operation 304, the first piece of metadata is stored in a common schema file. The common schema file has a format for defining and documenting a structure of a data model. At operation 306, the common schema file is bound to the delta table.

[0315] At operation 308, a request for a share of a range of the delta table is received from a client, the range including the delta table.

[0316] At operation 310, the latest common schema files for each of the delta tables in the range are aggregated into an aggregated common schema file. At operation 312, the aggregated common schema file is shared with the client.

[0317] In view of the above-described implementations of the subject matter, the following list of examples is disclosed, wherein one feature of an individual example or a combination of features of an individual example taken alone and, optionally, in combination with one or more features of one or more additional examples is an additional example that also falls within the disclosure of the present application:

[0318] Example 1 is a system comprising: at least one hardware processor; and a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: generating a first piece of metadata for a delta table in a data lake, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between multiple versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving a request from a client for a share of a range of the delta table, the range including the delta table; aggregating the latest common schema files for each of the delta tables in the range into an aggregated common schema file; and sharing the aggregated common schema file with the client.

[0319] In Example 2, the subject matter of Example 1 includes, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema files for each of the delta tables in the range.

[0320] In Example 3, the subject matter of Examples 1-2 includes, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file in the delta table in the range.

[0321] In Example 4, the subject matter of Example 3 includes, wherein the aggregating is asynchronously triggered.

[0322] In Example 5, the subject matter of Examples 1-4 includes, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the operations further comprise restoring consistency of the aggregated common schema file using the aggregation timestamp.

[0323] In Example 6, the subject matter of Examples 1-5 includes, wherein the first piece of metadata is used to train a machine learning model.

[0324] In Example 7, the subject matter of Examples 1-6 includes, wherein sharing is performed via a delta sharing application program interface (API).

[0325] Example 8 is a method comprising: generating a first piece of metadata for a delta table in a data lake, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between multiple versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving a request from a client for sharing of a scope of the delta table, the scope including the delta table; aggregating a latest common schema file for each delta table in the scope into an aggregated common schema file; and sharing the aggregated common schema file with the client.

[0326] In Example 9, the subject matter of Example 8 includes, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema file for each delta table in the scope.

[0327] In Example 10, the subject matter of Examples 8-9 includes, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file in the delta table in the scope.

[0328] In Example 11, the subject matter of Example 10 includes, wherein the aggregating is asynchronously triggered.

[0329] In Example 12, the subject matter of Examples 8-11 includes, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the method further comprises restoring a consistency of the aggregated common schema file using the aggregation timestamp.

[0330] In Example 13, the subject matter of Examples 8-12 includes, wherein the first piece of metadata is used to train a machine learning model.

[0331] In Example 14, the subject matter of Examples 8-13 includes, wherein sharing is performed via a delta sharing application program interface (API).

[0332] Example 15 is a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: generating a first piece of metadata for a delta table in a data lake, the data lake being a storage containing structured, unstructured, and semi-structured data, the delta table describing differences between a plurality of versions of a data class file stored in the data lake, the first piece of metadata having a first metadata type that is not supported by a format of the delta table; storing the first piece of metadata in a common schema file, the common schema file having a format for defining and documenting a structure of a data model; binding the common schema file to the delta table; receiving a request from a client for a share of a range of the delta table, the range including the delta table; aggregating a latest common schema file for each delta table in the range into an aggregated common schema file; and sharing the aggregated common schema file with the client.

[0333] In Example 16, the subject matter of Example 15 includes, wherein the aggregated common schema file represents a JavaScript Object Notation (JSON) merge of the latest common schema file for each delta table in the range.

[0334] In Example 17, the subject matter of Examples 15-16 includes, wherein the aggregating is performed in response to a write or delete operation that changes a state of the common schema file for a delta table in the range.

[0335] In Example 18, the subject matter of Examples 11-17 includes, wherein the aggregating is triggered asynchronously.

[0336] In Example 19, the subject matter of Examples 8-18 includes, wherein the aggregated common schema file contains an aggregation timestamp, and wherein the operations further comprise restoring a consistency of the aggregated common schema file using the aggregation timestamp.

[0337] In Example 20, the subject matter of Examples 8-19 includes, wherein the first piece of metadata is used to train a machine learning model.

[0338] Example 21 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0339] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0340] Example 23 is a system for implementing any of Examples 1-20.

[0341] Example 24 is a method for implementing any of Examples 1-20.

[0342] Figure 4 is a block diagram 400 illustrating a software architecture 402, which can be installed on any one or more of the devices described above. Figure 4 This is merely a non-limiting example of a software architecture, and it will be appreciated that many other architectures can be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 402 is implemented by the machine 500 of Fig. 1 having hardware 500 such as that illustrated in Fig. 2, which includes processors 510, memory 530, and input / output (I / O) components 550. In this example architecture, the software architecture 402 can be conceptualized as a stack of layers, where each layer can provide a particular functionality. Figure 5 Figure 4 The software architecture 402 of Fig. 1 can be conceptualized as a stack of layers, where each layer can provide a particular functionality. For example, the software architecture 402 includes layers such as an operating system 404, libraries 406, frameworks 408, and applications 410. In operation, consistent with some embodiments, the applications 410 invoke application programming interface (API) calls 412 through the software stack, and receive messages 414 in response to the API calls 412.

[0343] In various implementations, the operating system 404 manages hardware resources and provides common services. The operating system 404 includes, for example, a kernel 420, services 422, and drivers 424. Consistent with some embodiments, the kernel 420 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 420 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 422 can provide other common services that the other software layers can use. The drivers 424 are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 424 can include display drivers, camera drivers, Bluetooth® or Bluetooth® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth.

[0344] ​In some embodiments, libraries 406 provide a low-level common infrastructure used by the applications 410. Libraries 406 can include system libraries 430 (e.g., C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, libraries 406 can include API libraries 432 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics [PNG]), graphics libraries (e.g., an OpenGL framework used to render two dimensional (2D) and three dimensional (3D) graphics in an image context on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. Libraries 406 also include a wide variety of other libraries 434 to provide many other APIs to the applications 410.

[0345] The frameworks 408 provide a high-level common infrastructure used by the applications 410. For example, the frameworks 408 provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks 408 can provide a broad spectrum of other APIs that can be utilized by the applications 410, some of which can be specific to a particular operating system 404 or platform.

[0346] In example embodiments, the applications 410 include a home application 450, a contacts application 452, a browser application 454, a book reader application 456, a location application 458, a media application 460, a messaging application 462, a game application 464, and a broad assortment of other applications such as a third party application 466. The applications 410 are programs that execute functions defined in the programs. Various programming languages can be employed to create one or more of the applications 410, structured in a variety of manners, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, the third party application 466 (e.g., an application developed using the Android System™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) can be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or another mobile operating system. In this example, the third party application 466 can invoke the API calls 412 provided by the operating system 504 to facilitate the functionality described herein.

[0347] Figure 5 FIG. 16 illustrates a diagrammatic representation of a machine in the form of a computer system 500 within which a set of instructions, for causing the machine 500 to perform any one or more of the methodologies discussed herein, can be executed. In particular, Figure 5 FIG. 16 illustrates a diagrammatic representation of a machine in the form of a computer system 500 within which a set of instructions, for causing the machine 500 to perform any one or more of the methodologies discussed herein, can be executed. In particular, Figure 3 FIG. 16 illustrates a diagrammatic representation of a machine in the form of a computer system 500 within which a set of instructions, for causing the machine 500 to perform any one or more of the methodologies discussed herein, can be executed. In particular, Figures 1-3 The instructions 516 transform the general, non-programmed machine 500 into a particular machine 500 programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine 500 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 500 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 500 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 516, sequentially or otherwise, that specify actions to be taken by machine 500. Further, while only a single machine 500 is illustrated, the term "machine" shall also be taken to include a collection of machines 500 that individually or jointly execute the instructions 516 to perform any one or more of the methodologies discussed herein.

[0348] The machine 500 can include processors 510, memory 530, and I / O components 550, which can be configured to communicate with one another via a bus 502. In an example embodiment, the processors 510 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 512 and a processor 514 that can execute the instructions 516. The term "processor" is intended to include a multi-core processor that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions 516 Figure 5 Multiple processors 510 are shown, but the machine 500 can include a single processor 512 with single cores, a single processor 512 with multiple cores (e.g., a multi-core processor 512), multiple processors 512, 514 with single cores, multiple processors 512, 514 with multiple cores, or any combination thereof.

[0349] The storage 530 can include a main memory 532, a static memory 534, and a storage unit 536, each accessible to the processors 510, such as via a bus 502. The main memory 532, static memory 534, and storage unit 536 store the instructions 516 embodying any one or more of the methodologies or functions described herein. The instructions 516 can also reside, completely or partially, within the main memory 532, within the static memory 534, within the storage unit 536, within at least one of the processors 510 (e.g., within a cache of the processor), or any suitable combination thereof, during

[0350] The I / O components 550 can include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I / O components 550 that are included in the particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 550 can include many other components that are not shown in FIG. 5. The I / O components 550 are grouped as shown primarily to simplify the following discussion and are not Figure 5 In various example embodiments, the I / O components 550 can include output components 552 and input components 554. The output components 552 can include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 554 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0351] In further example embodiments, the I / O components 550 can include biometric components 556, motion components 558, environmental components 560, or positioning components 562, among a variety of other components. For example, biometric components 556 can include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components 558 can include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), or the like. The environmental components 560 can include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a surrounding physical environment. The positioning components 562 can include location sensor components (e.g., a Global Positioning System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), or the like.

[0352] Communication can be implemented using a wide variety of technologies. The I / O components 550 can include communication components 564 operable to couple the machine 500 to a network 580 or to devices 570 via coupling 582 and coupling 572, respectively. For example, the communication components 564 can include a network interface component or another suitable device to interface with a network 580. In further examples, the communication components 564 can include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth®components (e.g., Bluetooth®Low Energy), Wi-Fi®components, and other

[0353] Moreover, the communication components 564 can detect identifiers or include components operable to detect identifiers. For example, the communication components 564 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information can be derived via the communication components 564, such as location via internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that can indicate a particular location, and so forth.

[0354] The various memories (i.e., 530, 532, 534, and / or the memory of the processor 510) and / or the storage unit 536 can store one or more sets of instructions 516 and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 516), when executed by the processor 510, cause various operations to implement the disclosed embodiments.

[0355] As used herein, the terms "machine-storage medium," "device-storage medium," and "computer-storage medium" mean the same thing and can be used interchangeably. The terms "machine-storage medium," "device-storage medium," and "computer-storage medium" specifically exclude propagated signals and include both volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, or program instructions. The computer-readable medium, machine-readable medium, and device-readable medium are tangible and non-transitory. The terms "machine-storage medium," "device-storage medium," and "computer-storage medium" specifically exclude

[0356] In various example embodiments, one or more portions of network 580 can be a self-organizing network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WW AN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless local area network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 580 or a portion of network 580 can include a wireless or cellular network, and coupling 582 can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 582 can implement any of a variety of types of data transfer technology, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data Rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer technology.

[0357] The instructions 516 can be transmitted or received over the network 580 via the network interface device and utilizing any one of a number of well-known transfer protocols (e.g., HTTP). Similarly, the instructions 516 can be transmitted or received using a transmission medium via the coupling 572 (e.g., a peer-to-peer coupling). The term "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" shall be taken to include any intangible medium that is capable of storing, encoding, or carrying the instructions 516 for execution by the machine 500, and includes digital or analog communications signals or other intangible media to facilitate communication of such software. Hence, the terms "transmission medium" and "signal medium" shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0358] The terms "machine-readable medium," "computer-readable medium," and "device- readable medium" mean the same thing, and can be used interchangeably in this disclosure. The terms are defined to include both machine storage media and transmission media. Accordingly, the terms include both storage devices / media and carrier waves / modulated data signals. < / csn-doc-valid-from-delta-table-version-5> < / csn-doc-valid-from-delta-table-version-0> < / timestamp> < / timestamp> ) Create a new CSN document, the specified endpoint should be requested. For example: In this example, the X-SAP-Delta-Sharing-CSN-Table-Start-Version header is optional and can be provided to indicate the version of the delta table from which the CSN document is valid. Without a start version provided, the CSN document will be considered the default for the table and will apply to any version of the table until a new "versioned" CSN document is created for the table. The request payload can describe the table schema in the form of a valid CSN document, but the file storage component 102 neither enforces that the schema definition matches the actual table schema nor validates that the JSON document adheres to the CSN notation specification. The table schema definition in the form of a CSN document is the responsibility of the data producer. In example embodiments, the API endpoint can enforce that: 1. The request payload is a valid JavaScript Object Notation (JSON) document 2. There is a definition dictionary in the first level JSON object; and 3. The definition dictionary has a record whose key is exactly the shared table name or $self, and the value is a JSON object The user can retrieve the CSN metadata with the corresponding GET API endpoint, for example: The header X-SAP-Delta-Sharing-CSN-Table-Version is optional and can be used to request the CSN document for a specific version of the delta table. If not provided, the default CSN document for the table will be returned. In the case where the table does not have a default CSN document attached and the header is not provided, the request can fail with a 404 not found response. As a special value for this header, -1 indicates that the CSN corresponding to the highest start table version should be returned. To get all versions of a CSN entity, the parameter includeVersions=true can be provided. The response will be the default CSN document (if available) under the entity object of the table with an additional annotation field @DeltaSharing.csnVersions whose value is a JSON object that maps each delta table start version to the corresponding CSN.Example: {"definition": {"MyShareTable": {"kind": "entity", "elements": {"id": {"key": true, "type": "cds.UUID", "notNull": true}, "name": {"type": "cds.String"}}, "@DeltaSharing.csnVersions": {"0": {"definition": {"MyShareTable": {"kind": "entity", "elements": {"id": {"key": true, "type": "cds.UUID", "notNull": true}, "name": {"type": "cds.String", "length": 50}}}}}, "8": {"definition": {"MyShareTable": {"kind": "entity", "elements": {"id": {"key": true, "type": "cds.UUID", "notNull": true}, "name": {"type": "cds.String", "length": 50}, "price": {"type": "cds.Decimal", "precision": 16, "scale": 3}}}}},...}} The parameters includeVersions and X-SAP-Delta-Sharing-CSN-Table-Version are mutually exclusive, i.e., they cannot be used in combination. If the table has no default CSN associated with it, the response will not contain element objects under the entity object of the table. {"definition": {"MyShareTable": {"kind": "entity", "@DeltaSharing.csnVersions": {"0": {"definition": {"MyShareTable": {"kind": "entity", "elements": {"id": {"key": true, "type": "cds.UUID", "notNull": true},...}}}},...}}}} Similar to the PUT API, the DELETE API can be used to disassociate and remove the entity of a particular table. For example: The X-SAP-Delta-Sharing-CSN-Table-Start-Version header is optional, if not provided, the default CSN will be removed. Otherwise, the CSN associated with the start version will be removed. To enable delta sharing clients to consume shared tables with managed metadata, the delta sharing APIs can be enhanced to optionally return the CSN metadata in the response. More specifically, the query table metadata, read data from table, and read change data feed APIs can include additional fields as part of the metadata objects in the response.The CSN can be included if any of the following conditions are met: 1. The request indicates through a header that this is an incremental sharing client. 2. The corresponding share in the catalog 106 is marked with a publishCSN: true attribute. As an example, the request and response for a query table metadata operation that includes CSN table metadata would look like the following: Request: GET https: / / <instance-fqdn> / sharing / vl / shares / {share} / schemas / {schema} / tables / MyShareTable / metadata or GET https: / / <instance-fqdn> / sharing / vl / shares / {share} / schemas / {schema} / tables / MyShareTable / metadata Response: {"protocol": {"minReaderVersion": 1}} {"metadata": {"id": "f8d5c169-3d01-4ca3-ad9e-7dc3355aedb2", "format": {"provider": "parquet"}, "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\" "partitionColumns": ["date"], "@sap / delta-sharing / csn": {"definition": {"MyShareTable": {"kind": "entity", "elements": {"id": {"key": true, "type": "cds.UUID", "notNull": true}, "name": {"type": "cds.String"}, "age": {"type": "cds.Integer", "notNull": true}}}}}}} To enable users to get a consolidated view of the CSN metadata for a share, a new Get Share CSN API can be introduced to the catalog 106 and its response will contain a unified CSN document aggregating the CSN entities for all tables within the share. In example embodiments, the API endpoint URL can be referenced in the API resource definition of the ORD document for describing and discovering incremental sharing with enhanced and rich semantic from within the Unified Customer Landscape (UCL).A response might look like: {"definition": {"transport": {"category": "context"}, "accommodation": {"category": "context"}, "Transportation.Flight": {"category": "entity", "elements": {...}}, "Transportation.TrainStation": {"category": "entity", "elements": {...}}, "Accomodation.Hotel": {"category": "entity", "elements": {...}}}} In general, the shared aggregate CSN represents a deep JSON merge of the latest CSN documents of all tables within the share. However, to avoid conflicts between tables with the same name in different schemas, table definition names can be qualified with the name of the schema they belong to. Each schema must have its own record of definitions of category contexts. Note that when an entity name in any of the CSN documents of the tables being aggregated is $self, the variable will be resolved to the qualified identifier of the shared table. In addition to this, replacing and qualifying table definition names might not be enough, and so it can also adapt the associated target. The schema can also be extended for cases where CSN documents from multiple shares can be aggregated. In this case, qualified entity names in the aggregate CSN document might contain multiple levels of namespace. Shares are the highest level of entities, and they should be defined as services, which are essentially represented by the "context" part of the "service" category. For example: {"definition": {"Share1": {"category": "service", "@DeltaSharing.entity": "share"}, "Share2": {"category": "service", "@DeltaSharing.entity": "share"}, "Share1.Schema1": {"category": "context", "@DeltaSharing.entity": "schema"}, "Share2.Schema1": {"category": "context", "@DeltaSharing.entity": "schema"}, "Share1.Schema1.Table1": {"category": "entity", "elements": {...}}, "Share2.Schema1.Table1": {"category": "entity", "elements": {...}},...}} For performance reasons, the merge of the shared aggregate CSN should not happen at the time of a request to get the shared CSN API, but rather during the write / delete operations that change the state of the shared CSN metadata. This should reduce I / O and computation, as getting the shared CSN is expected to be called more frequently than write / delete operations.During the attachment or removal of CSN documents to or from any table of a share using the create or delete share table CSN API, or during the deletion of a share table using the delete share API, the HDL file will asynchronously trigger the merging of the aggregated CSN documents, which will eventually be consistent. Batch processing should be possible so that accumulated events / changes in the queue can be aggregated in a single task. In addition to this, the HDL file should also maintain support for concurrent writers for different tables of a share. To prevent a situation where the IMDG loses an asynchronous aggregation task, causing the shared CSN to permanently become inconsistent / stale, it is necessary to "persist" the fact that the shared CSN document is dirty / stale. This will be done by storing an aggregation event marker. Later, the asynchronous task will perform the aggregation, merge the result into the shared CSN file, and clear the event marker. The following algorithm summarizes what should happen when the HDL file receives a request for any of the following operations: create / update shared table CSN, delete shared table, or delete shared table CSN. 1. Persist aggregation event marker The aggregation event marker is basically an empty file, whose name should contain the "event" timestamp, and should be stored under the shared metadata prefix:... / . < / schema> < / share>

Claims

1. A system comprising: At least one hardware processor; and A computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations including: Generate the first metadata record for an incremental table in a data lake, which is a storage containing structured, unstructured, and semi-structured data. The incremental table describes the differences between multiple versions of data class files stored in the data lake. The first metadata record has a first metadata type that is not supported by the format of the incremental table. The first metadata is stored in a public schema symbol file, which has a format for defining and recording the structure of the data model; Bind the public schema symbol file to the incremental table; Receive a request from the client to share a range of the incremental table, the range including the incremental table; Aggregate the latest public schema symbol file for each incremental table in the range into the aggregated public schema symbol file; and The aggregated public schema symbol file is shared with the client.

2. The system according to claim 1, wherein, The aggregated public schema symbol file represents the JavaScript object symbol JSON merge of the latest public schema symbol file for each incremental table in the range.

3. The system according to claim 1, wherein, Aggregation is performed in response to write or delete operations that change the state of the public schema symbol file in the incremental table within the range.

4. The system according to claim 3, wherein, Aggregation is triggered asynchronously.

5. The system of claim 1, wherein the aggregated public schema symbol file contains an aggregated timestamp, and wherein the operation further includes using the aggregated timestamp to restore the consistency of the aggregated public schema symbol file.

6. The system of claim 1, wherein the first metadata is used to train a machine learning model.

7. The system according to claim 1, wherein, Sharing is performed via the Incremental Share API.

8. A method comprising: Generate the first metadata record for an incremental table in a data lake, which is a storage containing structured, unstructured, and semi-structured data. The incremental table describes the differences between multiple versions of data class files stored in the data lake. The first metadata record has a first metadata type that is not supported by the format of the incremental table. The first metadata is stored in a public schema symbol file, which has a format for defining and recording the structure of the data model; Bind the public schema symbol file to the incremental table; Receive a request from the client to share a range of the incremental table, the range including the incremental table; Aggregate the latest public schema symbol file for each incremental table in the range into an aggregated public schema symbol file; and The aggregated public schema symbol file is shared with the client.

9. The method according to claim 8, wherein, The aggregated public schema symbol file represents the JavaScript object symbol JSON merge of the latest public schema symbol file for each incremental table in the range.

10. The method according to claim 8, wherein, Aggregation is performed in response to write or delete operations that change the state of the public schema symbol file in the incremental table within the range.

11. The method according to claim 10, wherein, Aggregation is triggered asynchronously.

12. The method according to claim 8, wherein, The aggregated public schema symbol file contains an aggregated timestamp, and the method further includes using the aggregated timestamp to restore the consistency of the aggregated public schema symbol file.

13. The method according to claim 8, wherein, The first piece of metadata is used to train the machine learning model.

14. The method according to claim 8, wherein, Sharing is performed via the Incremental Share API.

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: Generate the first metadata record for an incremental table in a data lake, which is a storage containing structured, unstructured, and semi-structured data. The incremental table describes the differences between multiple versions of data class files stored in the data lake. The first metadata record has a first metadata type that is not supported by the format of the incremental table. The first metadata is stored in a public schema symbol file, which has a format for defining and recording the structure of the data model; Bind the public schema symbol file to the incremental table; Receive a request from the client to share a range of the incremental table, the range including the incremental table; Aggregate the latest public schema symbol file for each incremental table in the range into an aggregated public schema symbol file; and The aggregated public schema symbol file is shared with the client.

16. The non-transitory machine-readable medium according to claim 15, wherein, The aggregated public schema symbol file represents the JavaScript object symbol JSON merge of the latest public schema symbol file for each incremental table in the range.

17. The non-transitory machine-readable medium according to claim 15, wherein, Aggregation is performed in response to write or delete operations that change the state of the public schema symbol file in the incremental table within the range.

18. The non-transitory machine-readable medium according to claim 17, wherein, Aggregation is triggered asynchronously.

19. The non-transitory machine-readable medium according to claim 15, wherein, The aggregated public schema symbol file contains an aggregated timestamp, and the operation further includes using the aggregated timestamp to restore the consistency of the aggregated public schema symbol file.

20. The non-transitory machine-readable medium according to claim 15, wherein, The first piece of metadata is used to train the machine learning model.

Images